We've been getting a lot of questions about how the Internet Archive digitizes books.
-
We've been getting a lot of questions about how the Internet Archive digitizes books.
The short answer: page by page, by hand.
You may remember our viral 2021 video of Eliza Zhang scanning a book. That's still how we do it.
Meet Eliza, and learn how we scan books: https://blog.archive.org/2021/02/09/meet-eliza-zhang-book-scanner-and-viral-video-star/
-
We've been getting a lot of questions about how the Internet Archive digitizes books.
The short answer: page by page, by hand.
You may remember our viral 2021 video of Eliza Zhang scanning a book. That's still how we do it.
Meet Eliza, and learn how we scan books: https://blog.archive.org/2021/02/09/meet-eliza-zhang-book-scanner-and-viral-video-star/
@internetarchive as someone who has had to scan books many times in grad school…that machine is beautiful and I am immensely envious I never got to use one
-
We've been getting a lot of questions about how the Internet Archive digitizes books.
The short answer: page by page, by hand.
You may remember our viral 2021 video of Eliza Zhang scanning a book. That's still how we do it.
Meet Eliza, and learn how we scan books: https://blog.archive.org/2021/02/09/meet-eliza-zhang-book-scanner-and-viral-video-star/
weird, that there is no destructive way to scan automatically
-
Amazing work but please please please stop using MRC in PDFs.
@gloriouscow @internetarchive Whats MRC? Shouldn't those pdf be in pdf/A? Preferably in pfd/A-2a.
-
@gloriouscow @internetarchive Whats MRC? Shouldn't those pdf be in pdf/A? Preferably in pfd/A-2a.
MRC is a compression technique ("Mixed Raster Content").
It basically lifts the text off a graphic onto its own layer, stores it as 1bpp which can be compressed as such, then the underlying graphic, now complete with text-shaped holes in it, can then be compressed with jpeg or something.
the problem is whatever workflow IA uses to do this dials compression to all the way to maximum which results in 1bpp text over a smeary, unreadable mess
-
MRC is a compression technique ("Mixed Raster Content").
It basically lifts the text off a graphic onto its own layer, stores it as 1bpp which can be compressed as such, then the underlying graphic, now complete with text-shaped holes in it, can then be compressed with jpeg or something.
the problem is whatever workflow IA uses to do this dials compression to all the way to maximum which results in 1bpp text over a smeary, unreadable mess
here's an actual example.
the left is from a Google Books scan. The right is from IA:
imo this severely undermines IA's mission to preserve our history and there's a metric shit ton of content on IA that unfortunately probably needs to be reprocessed if not rescanned.
-
here's an actual example.
the left is from a Google Books scan. The right is from IA:
imo this severely undermines IA's mission to preserve our history and there's a metric shit ton of content on IA that unfortunately probably needs to be reprocessed if not rescanned.
notice that the "D" in "dealer" has become an O because MRC has the "Xerox Bug"
-
notice that the "D" in "dealer" has become an O because MRC has the "Xerox Bug"
so not only are the graphics ruined, but the text is ruined too. not a great situation.
-
so not only are the graphics ruined, but the text is ruined too. not a great situation.
@gloriouscow @internetarchive Thanks. OCR should only be used as invisible 2nd layer. pdf/A shouldn't allow this. Version 1 had recognition errors which were corrected in v2. 3 and 4 allow for extra content, which is not advisable since it's unclear if it's accessible in the future. 2a should be choosen because of the accessibility requirement. Screen readers can read them properly.
-
@gloriouscow @internetarchive I second this request.
@Thorsted @gloriouscow @internetarchive i am of a mixed opinion here. MRC is good for archival but definitely sucks for the end user. it would be very nice if user-facing PDFs could be just the mask + get jbig2ified + get an OCR layer... but perhaps that is a task better suited for shadow libraries
-
@Thorsted @gloriouscow @internetarchive i am of a mixed opinion here. MRC is good for archival but definitely sucks for the end user. it would be very nice if user-facing PDFs could be just the mask + get jbig2ified + get an OCR layer... but perhaps that is a task better suited for shadow libraries
@jj @Thorsted @internetarchive see my posts later. It sucks for archival too.
-
@jj @Thorsted @internetarchive see my posts later. It sucks for archival too.
@gloriouscow @Thorsted @internetarchive i saw them, yeah -- this is not fundamental to MRC, no? just the internet archive's particular use of it. were compression not at max it should be ok
-
@jj @Thorsted @internetarchive see my posts later. It sucks for archival too.
@gloriouscow @jj @internetarchive agreed. Terrible for archiving.
-
@internetarchive Oh wow. That's a pretty neat scanner. And I find it impossible to believe it's more efficient to rip books apart when a thing like this exists... They just didn't care enough to look.
@nazokiyoubinbou @internetarchive
The other process is several times faster and tries to keep copyright lawyers at bay for some time. Win-win for Amazon etc. …
https://www.404media.co/we-tracked-a-shipment-of-rare-books-it-ended-at-an-amazon-ai-training-facility/ -
@gloriouscow @Thorsted @internetarchive i saw them, yeah -- this is not fundamental to MRC, no? just the internet archive's particular use of it. were compression not at max it should be ok
@jj @gloriouscow @internetarchive my issue is when looking at the metadata there is an excessive amount of images to process.
-
@gloriouscow @Thorsted @internetarchive i saw them, yeah -- this is not fundamental to MRC, no? just the internet archive's particular use of it. were compression not at max it should be ok
@jj @Thorsted @internetarchive I just know I've never seen a jpeg change a letter
-
@nazokiyoubinbou @internetarchive
The other process is several times faster and tries to keep copyright lawyers at bay for some time. Win-win for Amazon etc. …
https://www.404media.co/we-tracked-a-shipment-of-rare-books-it-ended-at-an-amazon-ai-training-facility/@marcel @internetarchive I think actually ripping pages out takes longer than what I'm seeing here. Unless they have some super exact process to machine cut the books open without cutting out any letters, but what I had heard in the past was they employed people to tear them up and feed the pages in.
I mean just look at that video. Flip, press, flip, press, flip, press. It's fast and accurate.
But yes, they destroy them on purpose, which was... kind of my point. They could shred/etc even if they preserved them. Though I'm not clear why they can't give them to a charity or something (tax break!)
BTW you shared a paywall article. I've already heard about that though and was specifically referring to it as an example, but if I hadn't that wouldn't be a particularly helpful link...
-
We've been getting a lot of questions about how the Internet Archive digitizes books.
The short answer: page by page, by hand.
You may remember our viral 2021 video of Eliza Zhang scanning a book. That's still how we do it.
Meet Eliza, and learn how we scan books: https://blog.archive.org/2021/02/09/meet-eliza-zhang-book-scanner-and-viral-video-star/
@internetarchive@mastodon.archive.org As someone who used to work in a print shop, and has done my share scanning stuff from books, I desperately wish we'd had a scanner like that for the book copy jobs I did. Laying books open on a flatbed scanner is so tedious, and the repetitive motion of having to pick up the book to turn the page between each scan is exhausting. This would still be tedious, but a little easier only having to turn the page.
You can tell this machine was designed by someone with care for both the book and the operator. -
@jj @gloriouscow @internetarchive my issue is when looking at the metadata there is an excessive amount of images to process.
@Thorsted @gloriouscow @internetarchive this is not entirely my experience. for my
current work i'm looking at a bunch of (IA) MRC-encoded documents.pdfimagesdumps the foreground/background/masks which gives you very clean pages of text (as images) upon inverting the masks. it's three images per page but this is Finewhat i do find difficult to do though is recovering inline images, since MRC (or at least IA's use of it) does not attempt to extract those at all
-
@jj @Thorsted @internetarchive I just know I've never seen a jpeg change a letter
@gloriouscow @Thorsted @internetarchive fair point!
