The best time for a European organization to start building an independent European mirror of Wikipedia, not reliant on the US or WMF, was in December 2016.
-
@bvibber @davidgerard @luis_in_brief my back of the envelope guess would be about 3 million a year for an org to run a reliable dark archive and 8 million a year to actually start running something that you could visit (not edit). and those numbers assume the foundation is still doing its job next to what the eu side would be doing and assumes a redundant, non-US, sovereign cloud or self hosted solution.
Only gets more expensive after that.Cost goes into starting an org and hiring. Custom mirror engineering, immediate legal costs over content disputes and ddos defense.
-
@luis_in_brief why? In the worst case, there's probably more than one mirror to start one when the need arises
@Profpatsch @luis_in_brief there isnt even any mirror at all of the media files. No one wants to bear that cost.
-
@Profpatsch @luis_in_brief there isnt even any mirror at all of the media files. No one wants to bear that cost.
@Profpatsch @luis_in_brief the mirrors also largely exclude all the process behind what the audience sees. And that process is probably more important to mirror than the raw ‘ready’ latest version of ‘some’ of the content.
-
@luis_in_brief god, the cost of WMF-level infra. has anyone got an estimate of what it would cost to stand up and run? @bvibber do you have numbers you are able to say?
@davidgerard @luis_in_brief @bvibber Wikipedia, but decentralised!
-
@Profpatsch @luis_in_brief the mirrors also largely exclude all the process behind what the audience sees. And that process is probably more important to mirror than the raw ‘ready’ latest version of ‘some’ of the content.
@TheDJ
Do you mind if I ask what you mean with 'all the process'? -
@davidgerard @luis_in_brief @bvibber for a start one could do a dark archive ready as a fail-over, compare with arxiv https://blog.tib.eu/2025/05/14/protecting-science-tib-builds-dark-archive-for-arxiv/
@davidgerard @bvibber @gedankenstuecke yeah, there’s a lot of things that could be done very differently (and very incrementally) if you’re trying to make Wikipedia independently available rather than cloning WMF.
-
@luis_in_brief why? In the worst case, there's probably more than one mirror to start one when the need arises
@Profpatsch there are no deliberate, complete, formal mirrors that I know of. Archive might have one?
-
@bvibber @davidgerard @luis_in_brief my back of the envelope guess would be about 3 million a year for an org to run a reliable dark archive and 8 million a year to actually start running something that you could visit (not edit). and those numbers assume the foundation is still doing its job next to what the eu side would be doing and assumes a redundant, non-US, sovereign cloud or self hosted solution.
Only gets more expensive after that.@TheDJ @bvibber @davidgerard @luis_in_brief
I'd say the primary goal should be a read-only mirror. Once you've got that, you've got defense against WP going dark, becoming inaccessible, or having stuff deleted by political edict -- and can move forward with additional pieces (editability, development...) once any significant shoes drop.
You wouldn't need an org anything like the size of MediaWiki just for that. At a glance, I'd say it's within reach of a modestly wealthy individual or a small org.
-
@davidgerard @luis_in_brief @bvibber Wikipedia, but decentralised!
@davidgerard @bvibber @iain I would settle for “robustly and independently mirrored” right now.
-
@davidgerard @bvibber @iain I would settle for “robustly and independently mirrored” right now.
@luis_in_brief @davidgerard my strategy is downloading the text only archive every so often and putting it on my NAS, but that's not exactly a scalable solution
-
@davidgerard @luis_in_brief i do not have numbers offhand, but wmf annual budget is a strict upper bound
@bvibber @davidgerard also good news, traffic is falling off a cliff so even if your goal were to split traffic 50-50 with the official site (unlikely for a variety of reasons) that’s 20% less outbound bandwidth to pay for than this time last year 🤪
-
@Profpatsch @luis_in_brief there isnt even any mirror at all of the media files. No one wants to bear that cost.
@TheDJ @Profpatsch @luis_in_brief A read-only mirror with all pages and all pages' revisions (approximately 5-10 TB, including talk pages, user pages, etc) and thumbnails of all used files hosted locally would be a few hundred TB at most (the entirety of Wikimedia Commons is just ~1 PB <https://commons.wikimedia.org/wiki/Special:MediaStatistics>, and that's all files ever uploaded to Wikimedia servers with their original sizes and revision history); not very much.
-
@TheDJ @Profpatsch @luis_in_brief A read-only mirror with all pages and all pages' revisions (approximately 5-10 TB, including talk pages, user pages, etc) and thumbnails of all used files hosted locally would be a few hundred TB at most (the entirety of Wikimedia Commons is just ~1 PB <https://commons.wikimedia.org/wiki/Special:MediaStatistics>, and that's all files ever uploaded to Wikimedia servers with their original sizes and revision history); not very much.
@TheDJ @Profpatsch @luis_in_brief Getting the thumbnails will be a problem, because the last dump of all media files is 13 years old <https://ftpmirror.your.org/pub/wikimedia/images/wikipedia/commons/f/>, but I guess it's possible to just rip them off a "maxi" Kiwix dump. These only have files used in articles, though.
-
The best time for a European organization to start building an independent European mirror of Wikipedia, not reliant on the US or WMF, was in December 2016.
The second-best time is today.
@luis_in_brief Honestly I wouldn't even oppose mirrors hosted by the Wikimedia Foundation themselves. wikipedia.ph, wikipedia.is, wikipedia.gl, etc.
-
@luis_in_brief Honestly I wouldn't even oppose mirrors hosted by the Wikimedia Foundation themselves. wikipedia.ph, wikipedia.is, wikipedia.gl, etc.
@raulmatias no, that’s a worst-case scenario: exposes WMF to local laws/pressure, and keeps WMF as a single point of failure.
-
@luis_in_brief Honestly I wouldn't even oppose mirrors hosted by the Wikimedia Foundation themselves. wikipedia.ph, wikipedia.is, wikipedia.gl, etc.
@luis_in_brief Possibly with data being served from different data centers located in different countries for each domain.
-
@raulmatias no, that’s a worst-case scenario: exposes WMF to local laws/pressure, and keeps WMF as a single point of failure.
@luis_in_brief
>local laws/pressure.is is pretty much bulletproof, not sure about .ph and .gl.
-
@TheDJ @Profpatsch @luis_in_brief Getting the thumbnails will be a problem, because the last dump of all media files is 13 years old <https://ftpmirror.your.org/pub/wikimedia/images/wikipedia/commons/f/>, but I guess it's possible to just rip them off a "maxi" Kiwix dump. These only have files used in articles, though.
@raulmatias @Profpatsch @luis_in_brief yes, I know all this. 20+ year mediawiki dev here
-
The best time for a European organization to start building an independent European mirror of Wikipedia, not reliant on the US or WMF, was in December 2016.
The second-best time is today.
@luis_in_brief true
-
The best time for a European organization to start building an independent European mirror of Wikipedia, not reliant on the US or WMF, was in December 2016.
The second-best time is today.
@luis_in_brief Just as #OpenStreetMap Wikipedia is legally forkable but not in practical terms because you can't clone the contributor community. Without that you just end up with a stale copy.
