<![CDATA[We've been getting a lot of questions about how the Internet Archive digitizes books.]]>We've been getting a lot of questions about how the Internet Archive digitizes books.

The short answer: page by page, by hand.

You may remember our viral 2021 video of Eliza Zhang scanning a book. That's still how we do it.

Meet Eliza, and learn how we scan books: https://blog.archive.org/2021/02/09/meet-eliza-zhang-book-scanner-and-viral-video-star/

]]>https://forum.fedi.dk/topic/0fedc07c-9500-488c-82b8-33dc9d6786f9/we-ve-been-getting-a-lot-of-questions-about-how-the-internet-archive-digitizes-books.RSS for NodeFri, 11 Sep 2026 18:05:28 GMTTue, 25 Aug 2026 20:56:34 GMT60<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 06:35:11 GMT]]>@internetarchive

There's no need to destroy books in the process. The techbros just want to make it as cheap as possible, with the added benefit of taking books out of circulation so people need digital technology to access them. It's a retreat from the egalitarian printed book, back into a world where information is controlled by the wealthy. The reverse of progress.

]]>https://forum.fedi.dk/post/https://mastodon.social/users/riggbeck/statuses/117160418440955651https://forum.fedi.dk/post/https://mastodon.social/users/riggbeck/statuses/117160418440955651Wed, 26 Aug 2026 06:35:11 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 06:09:49 GMT]]>@internetarchive surely any book printed in the last 35 Years was made digitally in the first place? Do publishers really delete those? Or is there some weird copyright loophole for scanning a printed version?

]]>https://forum.fedi.dk/post/https://infosec.exchange/users/ketumbra/statuses/117160318687179790https://forum.fedi.dk/post/https://infosec.exchange/users/ketumbra/statuses/117160318687179790Wed, 26 Aug 2026 06:09:49 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 05:30:39 GMT]]>@Hammerwell @gloriouscow @internetarchive So in terms of "stop using MRC in PDFs", I guess the answer is "Done"

]]>https://forum.fedi.dk/post/https://mastodon.archive.org/users/textfiles/statuses/117160164696371049https://forum.fedi.dk/post/https://mastodon.archive.org/users/textfiles/statuses/117160164696371049Wed, 26 Aug 2026 05:30:39 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 05:29:46 GMT]]>@Hammerwell @gloriouscow @internetarchive And here is the .TIFF file that PDF was generated from.

https://archive.org/download/pc_magazine-1984_04_17/pc_magazine-1984_04_17.cbz/pc_magazine-1984_04_17%2Fpc_magazine-1984_04_17.294.tiff

]]>https://forum.fedi.dk/post/https://mastodon.archive.org/users/textfiles/statuses/117160161220658707https://forum.fedi.dk/post/https://mastodon.archive.org/users/textfiles/statuses/117160161220658707Wed, 26 Aug 2026 05:29:46 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 05:27:48 GMT]]>@deathkitten @internetarchive original design by Daniel Reetz: https://diybookscanner.org/archivist/

Sadly IA has not released design docs for their upgraded version, despite being asked repeatedly.

]]>https://forum.fedi.dk/post/https://mastodon.social/users/nightclaw/statuses/117160153480848206https://forum.fedi.dk/post/https://mastodon.social/users/nightclaw/statuses/117160153480848206Wed, 26 Aug 2026 05:27:48 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 05:27:27 GMT]]>@Hammerwell @gloriouscow @internetarchive OK, well, the file you are linking to (In https://archive.org/details/PCMAG) was not scanned by Internet Archive. It was uploaded by a user 5 years ago, using whatever PDF approach they decided to take, or, more likely, was taken by someone THEY duplicated the scans from.

Here's another non-IA-done scan, with proper work.

]]>https://forum.fedi.dk/post/https://mastodon.archive.org/users/textfiles/statuses/117160152079931740https://forum.fedi.dk/post/https://mastodon.archive.org/users/textfiles/statuses/117160152079931740Wed, 26 Aug 2026 05:27:27 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 05:27:17 GMT]]>@nazokiyoubinbou @internetarchive
From the descriptions I've seen, I imagine, they just cut the spine abd a few Millimeters off. There is some 2 cm of leeway there, exact cutting not required.

A quick product search revealed scanners with 3 sheets per second, no human operator required. From a "scan things fast and cheap" standpoint, that is much "better" than what's done here. (Of course, the IA approach has several benefits to the AI approach, but I don't need to point them out here.)

]]>https://forum.fedi.dk/post/https://waldvogel.family/users/marcel/statuses/117160151481078145https://forum.fedi.dk/post/https://waldvogel.family/users/marcel/statuses/117160151481078145Wed, 26 Aug 2026 05:27:17 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 04:50:47 GMT]]>@gloriouscow @Thorsted @internetarchive fair point!

]]>https://forum.fedi.dk/post/https://types.pl/users/jj/statuses/117160007925504927https://forum.fedi.dk/post/https://types.pl/users/jj/statuses/117160007925504927Wed, 26 Aug 2026 04:50:47 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 04:49:59 GMT]]>@Thorsted @gloriouscow @internetarchive this is not entirely my experience. for my
current work i'm looking at a bunch of (IA) MRC-encoded documents. pdfimages dumps the foreground/background/masks which gives you very clean pages of text (as images) upon inverting the masks. it's three images per page but this is Fine

what i do find difficult to do though is recovering inline images, since MRC (or at least IA's use of it) does not attempt to extract those at all

]]>https://forum.fedi.dk/post/https://types.pl/users/jj/statuses/117160004756197080https://forum.fedi.dk/post/https://types.pl/users/jj/statuses/117160004756197080Wed, 26 Aug 2026 04:49:59 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 04:46:58 GMT]]>@internetarchive@mastodon.archive.org As someone who used to work in a print shop, and has done my share scanning stuff from books, I desperately wish we'd had a scanner like that for the book copy jobs I did. Laying books open on a flatbed scanner is so tedious, and the repetitive motion of having to pick up the book to turn the page between each scan is exhausting. This would still be tedious, but a little easier only having to turn the page.

You can tell this machine was designed by someone with care for both the book and the operator.

]]>https://forum.fedi.dk/post/https://firetribe.org/notes/aqd6ez4y76https://forum.fedi.dk/post/https://firetribe.org/notes/aqd6ez4y76Wed, 26 Aug 2026 04:46:58 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 04:45:43 GMT]]>@marcel @internetarchive I think actually ripping pages out takes longer than what I'm seeing here. Unless they have some super exact process to machine cut the books open without cutting out any letters, but what I had heard in the past was they employed people to tear them up and feed the pages in.

I mean just look at that video. Flip, press, flip, press, flip, press. It's fast and accurate.

But yes, they destroy them on purpose, which was... kind of my point. They could shred/etc even if they preserved them. Though I'm not clear why they can't give them to a charity or something (tax break!)

BTW you shared a paywall article. I've already heard about that though and was specifically referring to it as an example, but if I hadn't that wouldn't be a particularly helpful link...

]]>https://forum.fedi.dk/post/https://urusai.social/users/nazokiyoubinbou/statuses/117159987999548285https://forum.fedi.dk/post/https://urusai.social/users/nazokiyoubinbou/statuses/117159987999548285Wed, 26 Aug 2026 04:45:43 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 04:43:22 GMT]]>@jj @Thorsted @internetarchive I just know I've never seen a jpeg change a letter

]]>https://forum.fedi.dk/post/https://oldbytes.space/users/gloriouscow/statuses/117159978764951375https://forum.fedi.dk/post/https://oldbytes.space/users/gloriouscow/statuses/117159978764951375Wed, 26 Aug 2026 04:43:22 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 04:43:14 GMT]]>@jj @gloriouscow @internetarchive my issue is when looking at the metadata there is an excessive amount of images to process.

]]>https://forum.fedi.dk/post/https://digipres.club/users/Thorsted/statuses/117159978224470930https://forum.fedi.dk/post/https://digipres.club/users/Thorsted/statuses/117159978224470930Wed, 26 Aug 2026 04:43:14 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 04:42:49 GMT]]>@nazokiyoubinbou @internetarchive
The other process is several times faster and tries to keep copyright lawyers at bay for some time. Win-win for Amazon etc. …
https://www.404media.co/we-tracked-a-shipment-of-rare-books-it-ended-at-an-amazon-ai-training-facility/

]]>https://forum.fedi.dk/post/https://waldvogel.family/users/marcel/statuses/117159976576734086https://forum.fedi.dk/post/https://waldvogel.family/users/marcel/statuses/117159976576734086Wed, 26 Aug 2026 04:42:49 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 04:41:44 GMT]]>@gloriouscow @jj @internetarchive agreed. Terrible for archiving.

]]>https://forum.fedi.dk/post/https://digipres.club/users/Thorsted/statuses/117159972346262023https://forum.fedi.dk/post/https://digipres.club/users/Thorsted/statuses/117159972346262023Wed, 26 Aug 2026 04:41:44 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 04:40:58 GMT]]>@gloriouscow @Thorsted @internetarchive i saw them, yeah -- this is not fundamental to MRC, no? just the internet archive's particular use of it. were compression not at max it should be ok

]]>https://forum.fedi.dk/post/https://types.pl/users/jj/statuses/117159969312926711https://forum.fedi.dk/post/https://types.pl/users/jj/statuses/117159969312926711Wed, 26 Aug 2026 04:40:58 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 04:39:00 GMT]]>@jj @Thorsted @internetarchive see my posts later. It sucks for archival too.

]]>https://forum.fedi.dk/post/https://oldbytes.space/users/gloriouscow/statuses/117159961594928786https://forum.fedi.dk/post/https://oldbytes.space/users/gloriouscow/statuses/117159961594928786Wed, 26 Aug 2026 04:39:00 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 04:35:55 GMT]]>@Thorsted @gloriouscow @internetarchive i am of a mixed opinion here. MRC is good for archival but definitely sucks for the end user. it would be very nice if user-facing PDFs could be just the mask + get jbig2ified + get an OCR layer... but perhaps that is a task better suited for shadow libraries

]]>https://forum.fedi.dk/post/https://types.pl/users/jj/statuses/117159949486620796https://forum.fedi.dk/post/https://types.pl/users/jj/statuses/117159949486620796Wed, 26 Aug 2026 04:35:55 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 04:24:18 GMT]]>@gloriouscow @internetarchive Thanks. OCR should only be used as invisible 2nd layer. pdf/A shouldn't allow this. Version 1 had recognition errors which were corrected in v2. 3 and 4 allow for extra content, which is not advisable since it's unclear if it's accessible in the future. 2a should be choosen because of the accessibility requirement. Screen readers can read them properly.

]]>https://forum.fedi.dk/post/https://troet.cafe/users/Hammerwell/statuses/117159903806572936https://forum.fedi.dk/post/https://troet.cafe/users/Hammerwell/statuses/117159903806572936Wed, 26 Aug 2026 04:24:18 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 04:16:45 GMT]]>@Hammerwell @internetarchive

so not only are the graphics ruined, but the text is ruined too. not a great situation.

]]>https://forum.fedi.dk/post/https://oldbytes.space/users/gloriouscow/statuses/117159874129740326https://forum.fedi.dk/post/https://oldbytes.space/users/gloriouscow/statuses/117159874129740326Wed, 26 Aug 2026 04:16:45 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 04:14:57 GMT]]>@Hammerwell @internetarchive

notice that the "D" in "dealer" has become an O because MRC has the "Xerox Bug"

https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning

]]>https://forum.fedi.dk/post/https://oldbytes.space/users/gloriouscow/statuses/117159867049662099https://forum.fedi.dk/post/https://oldbytes.space/users/gloriouscow/statuses/117159867049662099Wed, 26 Aug 2026 04:14:57 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 04:13:03 GMT]]>@Hammerwell @internetarchive

here's an actual example.

the left is from a Google Books scan. The right is from IA:

imo this severely undermines IA's mission to preserve our history and there's a metric shit ton of content on IA that unfortunately probably needs to be reprocessed if not rescanned.

]]>https://forum.fedi.dk/post/https://oldbytes.space/users/gloriouscow/statuses/117159859527547012https://forum.fedi.dk/post/https://oldbytes.space/users/gloriouscow/statuses/117159859527547012Wed, 26 Aug 2026 04:13:03 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 04:10:29 GMT]]>@Hammerwell @internetarchive

MRC is a compression technique ("Mixed Raster Content").

It basically lifts the text off a graphic onto its own layer, stores it as 1bpp which can be compressed as such, then the underlying graphic, now complete with text-shaped holes in it, can then be compressed with jpeg or something.

the problem is whatever workflow IA uses to do this dials compression to all the way to maximum which results in 1bpp text over a smeary, unreadable mess

]]>https://forum.fedi.dk/post/https://oldbytes.space/users/gloriouscow/statuses/117159849482028151https://forum.fedi.dk/post/https://oldbytes.space/users/gloriouscow/statuses/117159849482028151Wed, 26 Aug 2026 04:10:29 GMT<![CDATA[Reply to We've been getting a lot of questions about how the Internet Archive digitizes books. on Wed, 26 Aug 2026 04:06:40 GMT]]>@gloriouscow @internetarchive Whats MRC? Shouldn't those pdf be in pdf/A? Preferably in pfd/A-2a.

]]>https://forum.fedi.dk/post/https://troet.cafe/users/Hammerwell/statuses/117159834441866728https://forum.fedi.dk/post/https://troet.cafe/users/Hammerwell/statuses/117159834441866728Wed, 26 Aug 2026 04:06:40 GMT