We've been getting a lot of questions about how the Internet Archive digitizes books.
-
@gloriouscow @internetarchive I second this request.
@Thorsted @gloriouscow @internetarchive i am of a mixed opinion here. MRC is good for archival but definitely sucks for the end user. it would be very nice if user-facing PDFs could be just the mask + get jbig2ified + get an OCR layer... but perhaps that is a task better suited for shadow libraries
-
@Thorsted @gloriouscow @internetarchive i am of a mixed opinion here. MRC is good for archival but definitely sucks for the end user. it would be very nice if user-facing PDFs could be just the mask + get jbig2ified + get an OCR layer... but perhaps that is a task better suited for shadow libraries
@jj @Thorsted @internetarchive see my posts later. It sucks for archival too.
-
@jj @Thorsted @internetarchive see my posts later. It sucks for archival too.
@gloriouscow @Thorsted @internetarchive i saw them, yeah -- this is not fundamental to MRC, no? just the internet archive's particular use of it. were compression not at max it should be ok
-
@jj @Thorsted @internetarchive see my posts later. It sucks for archival too.
@gloriouscow @jj @internetarchive agreed. Terrible for archiving.
-
@internetarchive Oh wow. That's a pretty neat scanner. And I find it impossible to believe it's more efficient to rip books apart when a thing like this exists... They just didn't care enough to look.
@nazokiyoubinbou @internetarchive
The other process is several times faster and tries to keep copyright lawyers at bay for some time. Win-win for Amazon etc. …
https://www.404media.co/we-tracked-a-shipment-of-rare-books-it-ended-at-an-amazon-ai-training-facility/ -
@gloriouscow @Thorsted @internetarchive i saw them, yeah -- this is not fundamental to MRC, no? just the internet archive's particular use of it. were compression not at max it should be ok
@jj @gloriouscow @internetarchive my issue is when looking at the metadata there is an excessive amount of images to process.
-
@gloriouscow @Thorsted @internetarchive i saw them, yeah -- this is not fundamental to MRC, no? just the internet archive's particular use of it. were compression not at max it should be ok
@jj @Thorsted @internetarchive I just know I've never seen a jpeg change a letter
-
@nazokiyoubinbou @internetarchive
The other process is several times faster and tries to keep copyright lawyers at bay for some time. Win-win for Amazon etc. …
https://www.404media.co/we-tracked-a-shipment-of-rare-books-it-ended-at-an-amazon-ai-training-facility/@marcel @internetarchive I think actually ripping pages out takes longer than what I'm seeing here. Unless they have some super exact process to machine cut the books open without cutting out any letters, but what I had heard in the past was they employed people to tear them up and feed the pages in.
I mean just look at that video. Flip, press, flip, press, flip, press. It's fast and accurate.
But yes, they destroy them on purpose, which was... kind of my point. They could shred/etc even if they preserved them. Though I'm not clear why they can't give them to a charity or something (tax break!)
BTW you shared a paywall article. I've already heard about that though and was specifically referring to it as an example, but if I hadn't that wouldn't be a particularly helpful link...
-
We've been getting a lot of questions about how the Internet Archive digitizes books.
The short answer: page by page, by hand.
You may remember our viral 2021 video of Eliza Zhang scanning a book. That's still how we do it.
Meet Eliza, and learn how we scan books: https://blog.archive.org/2021/02/09/meet-eliza-zhang-book-scanner-and-viral-video-star/
@internetarchive@mastodon.archive.org As someone who used to work in a print shop, and has done my share scanning stuff from books, I desperately wish we'd had a scanner like that for the book copy jobs I did. Laying books open on a flatbed scanner is so tedious, and the repetitive motion of having to pick up the book to turn the page between each scan is exhausting. This would still be tedious, but a little easier only having to turn the page.
You can tell this machine was designed by someone with care for both the book and the operator. -
@jj @gloriouscow @internetarchive my issue is when looking at the metadata there is an excessive amount of images to process.
@Thorsted @gloriouscow @internetarchive this is not entirely my experience. for my
current work i'm looking at a bunch of (IA) MRC-encoded documents.pdfimagesdumps the foreground/background/masks which gives you very clean pages of text (as images) upon inverting the masks. it's three images per page but this is Finewhat i do find difficult to do though is recovering inline images, since MRC (or at least IA's use of it) does not attempt to extract those at all
-
@jj @Thorsted @internetarchive I just know I've never seen a jpeg change a letter
@gloriouscow @Thorsted @internetarchive fair point!
-
@marcel @internetarchive I think actually ripping pages out takes longer than what I'm seeing here. Unless they have some super exact process to machine cut the books open without cutting out any letters, but what I had heard in the past was they employed people to tear them up and feed the pages in.
I mean just look at that video. Flip, press, flip, press, flip, press. It's fast and accurate.
But yes, they destroy them on purpose, which was... kind of my point. They could shred/etc even if they preserved them. Though I'm not clear why they can't give them to a charity or something (tax break!)
BTW you shared a paywall article. I've already heard about that though and was specifically referring to it as an example, but if I hadn't that wouldn't be a particularly helpful link...
@nazokiyoubinbou @internetarchive
From the descriptions I've seen, I imagine, they just cut the spine abd a few Millimeters off. There is some 2 cm of leeway there, exact cutting not required.A quick product search revealed scanners with 3 sheets per second, no human operator required. From a "scan things fast and cheap" standpoint, that is much "better" than what's done here. (Of course, the IA approach has several benefits to the AI approach, but I don't need to point them out here.)
-
@gloriouscow @internetarchive Thanks. OCR should only be used as invisible 2nd layer. pdf/A shouldn't allow this. Version 1 had recognition errors which were corrected in v2. 3 and 4 allow for extra content, which is not advisable since it's unclear if it's accessible in the future. 2a should be choosen because of the accessibility requirement. Screen readers can read them properly.
@Hammerwell @gloriouscow @internetarchive OK, well, the file you are linking to (In https://archive.org/details/PCMAG) was not scanned by Internet Archive. It was uploaded by a user 5 years ago, using whatever PDF approach they decided to take, or, more likely, was taken by someone THEY duplicated the scans from.
Here's another non-IA-done scan, with proper work.
-
@internetarchive@mastodon.archive.org As someone who used to work in a print shop, and has done my share scanning stuff from books, I desperately wish we'd had a scanner like that for the book copy jobs I did. Laying books open on a flatbed scanner is so tedious, and the repetitive motion of having to pick up the book to turn the page between each scan is exhausting. This would still be tedious, but a little easier only having to turn the page.
You can tell this machine was designed by someone with care for both the book and the operator.@deathkitten @internetarchive original design by Daniel Reetz: https://diybookscanner.org/archivist/
Sadly IA has not released design docs for their upgraded version, despite being asked repeatedly.
-
@Hammerwell @gloriouscow @internetarchive OK, well, the file you are linking to (In https://archive.org/details/PCMAG) was not scanned by Internet Archive. It was uploaded by a user 5 years ago, using whatever PDF approach they decided to take, or, more likely, was taken by someone THEY duplicated the scans from.
Here's another non-IA-done scan, with proper work.
@Hammerwell @gloriouscow @internetarchive And here is the .TIFF file that PDF was generated from.
-
@Hammerwell @gloriouscow @internetarchive And here is the .TIFF file that PDF was generated from.
@Hammerwell @gloriouscow @internetarchive So in terms of "stop using MRC in PDFs", I guess the answer is "Done"
-
We've been getting a lot of questions about how the Internet Archive digitizes books.
The short answer: page by page, by hand.
You may remember our viral 2021 video of Eliza Zhang scanning a book. That's still how we do it.
Meet Eliza, and learn how we scan books: https://blog.archive.org/2021/02/09/meet-eliza-zhang-book-scanner-and-viral-video-star/
@internetarchive surely any book printed in the last 35 Years was made digitally in the first place? Do publishers really delete those? Or is there some weird copyright loophole for scanning a printed version?
-
We've been getting a lot of questions about how the Internet Archive digitizes books.
The short answer: page by page, by hand.
You may remember our viral 2021 video of Eliza Zhang scanning a book. That's still how we do it.
Meet Eliza, and learn how we scan books: https://blog.archive.org/2021/02/09/meet-eliza-zhang-book-scanner-and-viral-video-star/
There's no need to destroy books in the process. The techbros just want to make it as cheap as possible, with the added benefit of taking books out of circulation so people need digital technology to access them. It's a retreat from the egalitarian printed book, back into a world where information is controlled by the wealthy. The reverse of progress.
-
J jwcph@helvede.net shared this topic
