Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
-
i took a LIS course that talked about storage problems and longevity with microfiche and that was the story then, still the story now.
Standard practice among tape lib operators where I've worked: Multiple copies of each file/object. Min 3 matching reads from recent tapes + older tapes to create a new copy. Tapes regularly refreshed w/ new (usually higher capacity) cartridges.
Then along comes Orrible, "We are cancelling your tape orders. Move your data to our cloud where we'll keep one copy at much higher opex spend for you!"
This is not called doing business. Rather, it's doing business to you.
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands the thing is that this process is inherently brittle and myriad things can go wrong during copying, especially if the media is marginal. It’s inevitable that at least some loss is going to occur until we have a durable means to store data, and for older data rarely copied, it’s probable some could be lost forever. This is already happening with floppies.
This also presumes that the intersection of “someone or some people interested in preserving this data” and “someone or some people who are able to replace the old media” are the same people, which is also not guaranteed.
-
Standard practice among tape lib operators where I've worked: Multiple copies of each file/object. Min 3 matching reads from recent tapes + older tapes to create a new copy. Tapes regularly refreshed w/ new (usually higher capacity) cartridges.
Then along comes Orrible, "We are cancelling your tape orders. Move your data to our cloud where we'll keep one copy at much higher opex spend for you!"
This is not called doing business. Rather, it's doing business to you.
there was a period where tapes were slow/expensive enough and disk was cheap where lots of folks bought into RAID as good enough for one copy. problem was, many didn't do their homework about what the differences and risks were of the various types of RAID. also doesn't solve the "what happens if your primary site is destroyed".
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
I just finished up working on the Artemis 2 archiving (it’s under review now for the public October data release) — this is how nasa handles digital data for long long term archiving:
https://pds.nasa.gov/home/about/
And how long is long term archiving. Recall the Silurian hypothesis.
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
My current solution on my desktop is a btrfs partition with a bunch of random drives and a nightly job that rebalances the data across them if it needs to be (time boxed to 1 hour per night, restarts the night after it ends)
Its not quite zero touch but it does mean that if I replace a drive then it will slowly get redundant copies of data over the next few nights. The big failure mode is that it is not resistant to multiple drives failing before the first one is rebalanced
-
@inthehands the thing is that this process is inherently brittle and myriad things can go wrong during copying, especially if the media is marginal. It’s inevitable that at least some loss is going to occur until we have a durable means to store data, and for older data rarely copied, it’s probable some could be lost forever. This is already happening with floppies.
This also presumes that the intersection of “someone or some people interested in preserving this data” and “someone or some people who are able to replace the old media” are the same people, which is also not guaranteed.
@inthehands an example of how brittle this process is:
Most of Google Video is essentially gone forever. So is most of MySpace and countless irreplaceable songs. The main reason being that those interested in preserving it and those with the ability to preserve it just… did not intersect, which is the fate that usually happens to most data. Usenet’s archives are also very incomplete.
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands "Keep circulating the tapes."
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands
You’re right about it being a process and a commitment rather than a physical thing. But in terms of physical things, I’m guessing that the most successful, proven long-term storage media we have are still illuminated manuscripts on bound parchment. Cuneiform tablets and oracle bones have held up well too. -
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands
Binary in dots printed into Titanium (a company tried Iridium) might come moderately close, perhaps?The Eternity Service as described by Ross Anderson is another element, protecting against loss of a physical copy archive, and against some other attacks.
#Scifi The trouble in #VernorVinge 's #AFireAcrossTheDeep SF novel, one of the two set in the #ZonesOfMind begins with Stromli Realm's team opening an ancient archive they discovered in the Low Beyond...
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands @overholt Yup, I’ve been a “Digital Archivist” for 30 years, personally and professionally. Only the process will save you. All media fails.
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands IMHO i think of preserval as a biological thing: if it doesn’t get reproduced, it dies. people think because it’s digital it’s like etching it in stone when it’s far from that
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands agreed.
The question is what happens when we won't be able to afford renewing the medium. Who gets to choose what to keep and what to loose? -
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands Kodachrome is eternal.
-
@inthehands an example of how brittle this process is:
Most of Google Video is essentially gone forever. So is most of MySpace and countless irreplaceable songs. The main reason being that those interested in preserving it and those with the ability to preserve it just… did not intersect, which is the fate that usually happens to most data. Usenet’s archives are also very incomplete.
a large amount of British TV aimed at teens/young adults, especially that which contained the slightest bit of "pop music" from *1990s/00s* is likely permanently gone.
Not for lack of archiving technology (it was around in 1990s, including digital video storage by late 90s, but because there was so much arguing over over copyright that archives have been allowed to deteriorate or even deliberately bulk erased to prevent them falling into "the wrong hands" ( TV companies would rather destroy it than risk a lawsuit because hoarded master tapes ended up on YouTube)
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
Quote from #SciAm article on this topic back in the 1990s/early 2000s...
"Digital information lasts forever--or five years, whichever comes first." - Jeff Rothenberg
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands @darius smiling placidly in agreement and screaming internally as I remember yet again the 2 SyQuest 40MB bernoulli discs sitting in my garage
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands For audio, wax cylinders and gramophone records (and vinyl) have outlived all other media and look set to do so for centuries to come. Libraries and archives have tons of media that is essentially unplayable without the appropriate defunct hardware.
-
a large amount of British TV aimed at teens/young adults, especially that which contained the slightest bit of "pop music" from *1990s/00s* is likely permanently gone.
Not for lack of archiving technology (it was around in 1990s, including digital video storage by late 90s, but because there was so much arguing over over copyright that archives have been allowed to deteriorate or even deliberately bulk erased to prevent them falling into "the wrong hands" ( TV companies would rather destroy it than risk a lawsuit because hoarded master tapes ended up on YouTube)
@vfrmedia @inthehands You often can't find copies of shows even into the 00's because of this. And also pretty much all early-streaming stuff is gone with no copies (never recorded, not many people *could* due to space, and DRM actively prevented it at the time). Same with a lot of early-Internet video. A lot of early YouTube is also gone.
-
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
We need the equivalent of stone tablets.
3D-printing using ECCs onto some kind of ceramic alloy that's super hard and super temperature resistant. -
RE: https://glammr.us/@overholt/117049333496122205
Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
@inthehands
There were some real research systems looking into this, and it makes me incredibly sad that they basically have all stopped because it's not immediately profitable.Living archives are always the functional solution, but it should still be something people work on!
The concept of immutable records should at least be appealing....
