We've been getting a lot of questions about how the Internet Archive digitizes books.
-
Amazing work but please please please stop using MRC in PDFs.
@gloriouscow @internetarchive Whats MRC? Shouldn't those pdf be in pdf/A? Preferably in pfd/A-2a.
-
@gloriouscow @internetarchive Whats MRC? Shouldn't those pdf be in pdf/A? Preferably in pfd/A-2a.
MRC is a compression technique ("Mixed Raster Content").
It basically lifts the text off a graphic onto its own layer, stores it as 1bpp which can be compressed as such, then the underlying graphic, now complete with text-shaped holes in it, can then be compressed with jpeg or something.
the problem is whatever workflow IA uses to do this dials compression to all the way to maximum which results in 1bpp text over a smeary, unreadable mess
-
MRC is a compression technique ("Mixed Raster Content").
It basically lifts the text off a graphic onto its own layer, stores it as 1bpp which can be compressed as such, then the underlying graphic, now complete with text-shaped holes in it, can then be compressed with jpeg or something.
the problem is whatever workflow IA uses to do this dials compression to all the way to maximum which results in 1bpp text over a smeary, unreadable mess
here's an actual example.
the left is from a Google Books scan. The right is from IA:
imo this severely undermines IA's mission to preserve our history and there's a metric shit ton of content on IA that unfortunately probably needs to be reprocessed if not rescanned.
-
here's an actual example.
the left is from a Google Books scan. The right is from IA:
imo this severely undermines IA's mission to preserve our history and there's a metric shit ton of content on IA that unfortunately probably needs to be reprocessed if not rescanned.
notice that the "D" in "dealer" has become an O because MRC has the "Xerox Bug"
-
notice that the "D" in "dealer" has become an O because MRC has the "Xerox Bug"
so not only are the graphics ruined, but the text is ruined too. not a great situation.
-
so not only are the graphics ruined, but the text is ruined too. not a great situation.
@gloriouscow @internetarchive Thanks. OCR should only be used as invisible 2nd layer. pdf/A shouldn't allow this. Version 1 had recognition errors which were corrected in v2. 3 and 4 allow for extra content, which is not advisable since it's unclear if it's accessible in the future. 2a should be choosen because of the accessibility requirement. Screen readers can read them properly.
-
@gloriouscow @internetarchive I second this request.
@Thorsted @gloriouscow @internetarchive i am of a mixed opinion here. MRC is good for archival but definitely sucks for the end user. it would be very nice if user-facing PDFs could be just the mask + get jbig2ified + get an OCR layer... but perhaps that is a task better suited for shadow libraries
-
@Thorsted @gloriouscow @internetarchive i am of a mixed opinion here. MRC is good for archival but definitely sucks for the end user. it would be very nice if user-facing PDFs could be just the mask + get jbig2ified + get an OCR layer... but perhaps that is a task better suited for shadow libraries
@jj @Thorsted @internetarchive see my posts later. It sucks for archival too.
-
@jj @Thorsted @internetarchive see my posts later. It sucks for archival too.
@gloriouscow @Thorsted @internetarchive i saw them, yeah -- this is not fundamental to MRC, no? just the internet archive's particular use of it. were compression not at max it should be ok
-
@jj @Thorsted @internetarchive see my posts later. It sucks for archival too.
@gloriouscow @jj @internetarchive agreed. Terrible for archiving.
-
@internetarchive Oh wow. That's a pretty neat scanner. And I find it impossible to believe it's more efficient to rip books apart when a thing like this exists... They just didn't care enough to look.
@nazokiyoubinbou @internetarchive
The other process is several times faster and tries to keep copyright lawyers at bay for some time. Win-win for Amazon etc. …
https://www.404media.co/we-tracked-a-shipment-of-rare-books-it-ended-at-an-amazon-ai-training-facility/ -
@gloriouscow @Thorsted @internetarchive i saw them, yeah -- this is not fundamental to MRC, no? just the internet archive's particular use of it. were compression not at max it should be ok
@jj @gloriouscow @internetarchive my issue is when looking at the metadata there is an excessive amount of images to process.
-
@gloriouscow @Thorsted @internetarchive i saw them, yeah -- this is not fundamental to MRC, no? just the internet archive's particular use of it. were compression not at max it should be ok
@jj @Thorsted @internetarchive I just know I've never seen a jpeg change a letter
-
@nazokiyoubinbou @internetarchive
The other process is several times faster and tries to keep copyright lawyers at bay for some time. Win-win for Amazon etc. …
https://www.404media.co/we-tracked-a-shipment-of-rare-books-it-ended-at-an-amazon-ai-training-facility/@marcel @internetarchive I think actually ripping pages out takes longer than what I'm seeing here. Unless they have some super exact process to machine cut the books open without cutting out any letters, but what I had heard in the past was they employed people to tear them up and feed the pages in.
I mean just look at that video. Flip, press, flip, press, flip, press. It's fast and accurate.
But yes, they destroy them on purpose, which was... kind of my point. They could shred/etc even if they preserved them. Though I'm not clear why they can't give them to a charity or something (tax break!)
BTW you shared a paywall article. I've already heard about that though and was specifically referring to it as an example, but if I hadn't that wouldn't be a particularly helpful link...
-
We've been getting a lot of questions about how the Internet Archive digitizes books.
The short answer: page by page, by hand.
You may remember our viral 2021 video of Eliza Zhang scanning a book. That's still how we do it.
Meet Eliza, and learn how we scan books: https://blog.archive.org/2021/02/09/meet-eliza-zhang-book-scanner-and-viral-video-star/
@internetarchive@mastodon.archive.org As someone who used to work in a print shop, and has done my share scanning stuff from books, I desperately wish we'd had a scanner like that for the book copy jobs I did. Laying books open on a flatbed scanner is so tedious, and the repetitive motion of having to pick up the book to turn the page between each scan is exhausting. This would still be tedious, but a little easier only having to turn the page.
You can tell this machine was designed by someone with care for both the book and the operator. -
@jj @gloriouscow @internetarchive my issue is when looking at the metadata there is an excessive amount of images to process.
@Thorsted @gloriouscow @internetarchive this is not entirely my experience. for my
current work i'm looking at a bunch of (IA) MRC-encoded documents.pdfimagesdumps the foreground/background/masks which gives you very clean pages of text (as images) upon inverting the masks. it's three images per page but this is Finewhat i do find difficult to do though is recovering inline images, since MRC (or at least IA's use of it) does not attempt to extract those at all
-
@jj @Thorsted @internetarchive I just know I've never seen a jpeg change a letter
@gloriouscow @Thorsted @internetarchive fair point!
-
@marcel @internetarchive I think actually ripping pages out takes longer than what I'm seeing here. Unless they have some super exact process to machine cut the books open without cutting out any letters, but what I had heard in the past was they employed people to tear them up and feed the pages in.
I mean just look at that video. Flip, press, flip, press, flip, press. It's fast and accurate.
But yes, they destroy them on purpose, which was... kind of my point. They could shred/etc even if they preserved them. Though I'm not clear why they can't give them to a charity or something (tax break!)
BTW you shared a paywall article. I've already heard about that though and was specifically referring to it as an example, but if I hadn't that wouldn't be a particularly helpful link...
@nazokiyoubinbou @internetarchive
From the descriptions I've seen, I imagine, they just cut the spine abd a few Millimeters off. There is some 2 cm of leeway there, exact cutting not required.A quick product search revealed scanners with 3 sheets per second, no human operator required. From a "scan things fast and cheap" standpoint, that is much "better" than what's done here. (Of course, the IA approach has several benefits to the AI approach, but I don't need to point them out here.)
-
@gloriouscow @internetarchive Thanks. OCR should only be used as invisible 2nd layer. pdf/A shouldn't allow this. Version 1 had recognition errors which were corrected in v2. 3 and 4 allow for extra content, which is not advisable since it's unclear if it's accessible in the future. 2a should be choosen because of the accessibility requirement. Screen readers can read them properly.
@Hammerwell @gloriouscow @internetarchive OK, well, the file you are linking to (In https://archive.org/details/PCMAG) was not scanned by Internet Archive. It was uploaded by a user 5 years ago, using whatever PDF approach they decided to take, or, more likely, was taken by someone THEY duplicated the scans from.
Here's another non-IA-done scan, with proper work.
-
@internetarchive@mastodon.archive.org As someone who used to work in a print shop, and has done my share scanning stuff from books, I desperately wish we'd had a scanner like that for the book copy jobs I did. Laying books open on a flatbed scanner is so tedious, and the repetitive motion of having to pick up the book to turn the page between each scan is exhausting. This would still be tedious, but a little easier only having to turn the page.
You can tell this machine was designed by someone with care for both the book and the operator.@deathkitten @internetarchive original design by Daniel Reetz: https://diybookscanner.org/archivist/
Sadly IA has not released design docs for their upgraded version, despite being asked repeatedly.
