Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
-
My current solution on my desktop is a btrfs partition with a bunch of random drives and a nightly job that rebalances the data across them if it needs to be (time boxed to 1 hour per night, restarts the night after it ends)
Its not quite zero touch but it does mean that if I replace a drive then it will slowly get redundant copies of data over the next few nights. The big failure mode is that it is not resistant to multiple drives failing before the first one is rebalanced
@gbargoud @inthehands But isn't the fact that it's in your home another potential failure mode? A house fire can undo all your careful rebalancing in an hour. (Of course, you don't mention offsite storage; you could have a copy at your sister's or something.)
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
I did the same for the Alberta Provincial Archives.
Same conclusion.
Most of their material was on microfilm, which was beginning to perish. Early 1980s.
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands - I remember a conversation with Ciaran McGinley at the EPO about his push for continuous media cycling to ensure that readers still functioned for the chosen medium.
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands LOCKSS (Lots of Copies Keep Stuff Safe) tries to mitigate the problem of integrity when copying files forward. TL:DR - min 4 copies with continuous file integrity checks.
(edit: it's mentioned in the Gizmodo article)
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands data storage is the easy part. Being able to decode the stored data long after the computers that wrote it are obsolete and no machine is available that can run the old binaries is the fun part. Even if you have carefully forward copied all the bits onto media you can read, the bits are just the starting point.
-
@gbargoud @inthehands But isn't the fact that it's in your home another potential failure mode? A house fire can undo all your careful rebalancing in an hour. (Of course, you don't mention offsite storage; you could have a copy at your sister's or something.)
Yes that is true but that's the same failure mode of every single site storage system, including libraries as in the original quoted post. Having additional redundancy in multiple locations is important for some information but each location needs to have that kind of maintenance happening for properly long term storage
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands Related: https://www.lockss.org/
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands the quipy way I've heard this described is that in archival efforts, you dont have "storage," you have "movage."
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
My preservation strategy is, no joke, a Nextcloud server that backs itself up on top of the data it contains - then pushes it to my desktop and laptop and partly to my smartphone. Bam, triple redundancy -
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands I've worked in numerical banks that archive their records. They generally use high quality optical storage. Gold or silver based cdr or mdisk. Both thought to last 30-100 years
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands
That's totally true. I watched a French documentary on long term nuclear waste storage sites which have to keep them for many centuries (millenia even!) and they were researching solutions to keep instructions on how to (not) handle and keep away from the site for future generations (civilisations even) who won't even speak speak the same language. They chose acid free paper in climate controlled rooms for the first decades and century. They were also speaking about genetically engineering a new bread of cats to glow near nuclear material and had some idea on songs about being fearful of glowing cats because it meant danger. They say that's how stories travel through millenias and that could help future people to not access the dangerous zones of the site, even after a civilisation collapse. Far fetched but interesting. -
@inthehands I've worked in numerical banks that archive their records. They generally use high quality optical storage. Gold or silver based cdr or mdisk. Both thought to last 30-100 years
@Eurospoofer @inthehands You still have to solve the availability of reading hardware and software in 100 years.
-
@inthehands @overholt Yup, I’ve been a “Digital Archivist” for 30 years, personally and professionally. Only the process will save you. All media fails.
@JustinDerrick definitly so. @inthehands @overholt
-
We need the equivalent of stone tablets.
3D-printing using ECCs onto some kind of ceramic alloy that's super hard and super temperature resistant.@megatronicthronbanks @inthehands The trouble with extremely durable high-density media is that reading them still requires sophisticated machines. In 100 years time, you probably won't find one that works, nor instructions for building one.
-
@inthehands My very non-technical historian husband says you should have multiple types of media and multiple locations.
@ClimateJenny @inthehands On different tectonic plates.
-
@megatronicthronbanks @inthehands The trouble with extremely durable high-density media is that reading them still requires sophisticated machines. In 100 years time, you probably won't find one that works, nor instructions for building one.
@mansr @inthehands What if it's just barcodes?
-
@mansr @inthehands What if it's just barcodes?
@megatronicthronbanks @inthehands How are you going to store the instructions for decoding them?
-
@megatronicthronbanks @inthehands How are you going to store the instructions for decoding them?
@mansr @inthehands Why would you need to? You only need to describe the format.
It's not a DVD or DLT, that requires precise physics to decode.
-
@mansr @inthehands Why would you need to? You only need to describe the format.
It's not a DVD or DLT, that requires precise physics to decode.
@megatronicthronbanks @inthehands So where will you store the description then? You can't expect anyone to just look at a bar/2D code and infer the arithmetic required to decode it.
-
@megatronicthronbanks @inthehands So where will you store the description then? You can't expect anyone to just look at a bar/2D code and infer the arithmetic required to decode it.
@mansr @inthehands In a few human languages, on pages scattered throughout to protect them. With diagrams.
We did it with a gold record, and that was for aliens.
