This is FASCINATING.
-
@SteveBellovin @mattblaze @hzulla brb writing a horror novel called “Children of the Cron.”
@20002ist @SteveBellovin @mattblaze @hzulla Sequel to _Crop Circles In the Korn Shell_?
-
@hzulla more for constraints on local compute capacity than the global power grid, but I can't be the only sysadmin who routinely staggers recurring jobs likely updates or backups to kick off at semi-random times like 02:17, 03:43, etc., based on exactly this sort of reasoning…
-
@hzulla Cron, of course, dates back to a world in which:
- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational loadSoftware often outlives all sorts of assumptions!
@mattblaze @hzulla i was thinking of this earlier after reading the news. Cron is now harming the grid because of course we didn’t think of that. Haha
-
The Linux Cron doesn't have a -J / -j option?
Sorry, I'm too lazy to look that up, but that's what I'm using on BSD. Not because I care for the power-grid, but It keeps load-spikes at bay (hello, Disk-IO)
-
@gabrielesvelto @benjamineskola @funkylab @hzulla I don't think any of the cron implementations that I've used have implemented RANDOM_DELAY across many *nixes.
RANDOM_DELAY is part of the POSIX standard, so I actually would be very, very surprised, if they don't.
Cronie, anachon, and even systemd supports it.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
Interesting. I've always assumed this and im allergic to :00 times. Probably i absorbed this from time immemorial and forgot it.
It also seems obvious. Not for power spikes, but everything else spikes, cpu etc.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla I asked the question about 4 years ago in the Home Assistant Reddit: https://www.reddit.com/r/homeassistant/s/rgqYwtLDZH
-
UK tv broadcasters had to deliberately stagger their commercial breaks as everyone put the kettle on to make tea during the commercials. And none of that non-British 1kW kettle nonsense, proper British kettles are 3kW!
@maya_b
Got this from a Danish power plant operator many years ago: They were of course not allowed TV or anything distracting in the control room, so they couldn't watch the big football internationals. They still knew the result as soon as the game ended.If Denmark had won, people would watch the TV a bit longer, enjoying the moment. If Denmark had lost, everybody would turn off, get up and put on the coffee machine right away. The load spike was noticeable.
@hzulla -
@amethyst @veronica @hzulla same here - active monitoring would create concurrent spikes in system load (which we don't archive over months) and thus in job runtime (which is kept for analysis) not really visible with the naked eye, but looking at the latter using resampling and fft made it really visible. We moved that part of the active monitoring to a later point in the script and selected a different start time in all calling offices...
Yeah, obvious in hindsight. -
@SteveBellovin @mattblaze @hzulla In the late 1990s I worked at a place that had a hundred or so systems all using the same file server. We used a simple wrapper for cron jobs run by all systems to smear out the impact: extract low octet of IP address, sleep that many seconds before starting.
Nowadays I manage about 300 systems, but net and CPU and storage are fast enough that we don't get crowd-load effects. If we did I'd repeat the scheme but sleep N/10 or N/100 seconds.
There's something to be said for changing cron to insert small random delays. I don't think it often (if ever) matters if a job starts a fraction of a second, or even a few seconds, after the exact start of the minute.
@oclsc @SteveBellovin @mattblaze @hzulla nice trick delaying based on the last octet.
Though I might add some other time / date attribute to the mix so lower IP doesn’t always beat higher IP.
But I like the base idea, a lot.
-
UK tv broadcasters had to deliberately stagger their commercial breaks as everyone put the kettle on to make tea during the commercials. And none of that non-British 1kW kettle nonsense, proper British kettles are 3kW!
-
Yes, in the USA, we have 120V through 14g sized wire, which is good enough for about 15 amps. I measure about 1600 watts for devices that claim 1800 watts. Some outlets are rated for 20 amps and those should be wired with 12g sized wire, or property damage is much more likely. For the stoves and clothes dryers, 10g sized wire is used at minimum, rated for 30 amps.
Imagine if all car chargers started at EXACTLY the same hundredth of a second, on the same rising phase of the waveform. You would be able hear the power company turbines ring like a bell from the magnetic hammer.
-
Exactly. Jitter option (-j for users, -J for root) was introduced in cron in FreeBSD 5.3 (late 1993) in order to address system load spikes. The grid effect was probably not evaluated at the time, because CPU/RAM/Network were scarce back then, compared to electricity.
-
@hzulla Cron, of course, dates back to a world in which:
- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational loadSoftware often outlives all sorts of assumptions!
@mattblaze @hzulla Which Cron? I mean there are, for good reasons, many different versions of Cron out there. There are Crons that target specific use cases the original writers didn't consider (like having computers that are turned off or in standby from time to time).
That's the wonderful thing about Free Software. It's not a "product" but instead a chorus of competing ideas and forks which explore new things.
-
RANDOM_DELAY is part of the POSIX standard, so I actually would be very, very surprised, if they don't.
Cronie, anachon, and even systemd supports it.
@shaknais Even if it were available, it seems to randomise by whole minutes, so not potentially easing the problem in this article.
"The RANDOM_DELAY variable allows delaying job startups by random amount of minutes with upper limit specified by the variable."
My current server cron is Vixie, which does not support this feature.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla I have worked on videogames my entire software engineering career. I have thought about the effects of my code on the powergrid. But more in terms of “if my code is inefficient it wastes power, if a million players play the game for so and so hours on their playstation that will amount to so and so many watts-ish”.
-
@datenwolf @hzulla Such an attack is more feasible with a relatively small number of devices if you attack the links between larger electricity networks.
Here's an article from 2017 (German only)
https://www.heise.de/news/Botnetze-koennen-das-Stromnetz-sabotieren-3927886.html?seite=all@newstik @datenwolf @hzulla the original paper is in English
Grid Shock: Coordinated Load-Changing Attacks on Power Grids
https://publications.sba-research.org/publications/201712%20-%20ADabrowski%20-%20Grid%20Shock.pdf
-
@hzulla Cron, of course, dates back to a world in which:
- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational loadSoftware often outlives all sorts of assumptions!
@mattblaze @hzulla I think we need to add "Software often outlives all sorts of assumptions" to one of the "Falsehoods programmers believe about…" awesome lists – perhaps under "Falsehoods programmers believe about time (yes, even astronomers)"…
-
@TimWardCam @hzulla Early adopters are unlikely to bring down the grid by themselves, and the big VPP players are expected to avoid herding also. This is for mainstream timed use, to smear those times.
I haven't run HA, but inserting even a few seconds randomly before bringing on a timed big load will reduce demand slew rate, so I do it routinely, as I have for servers. Random numbers and primes are two of my favourite robustness tools.
@DamonHD @TimWardCam @hzulla My favourite solution is to forget to enable the NTP server.
-
@SteveBellovin @mattblaze @hzulla brb writing a horror novel called “Children of the Cron.”
