This is FASCINATING.
-
@amethyst @veronica @hzulla same here - active monitoring would create concurrent spikes in system load (which we don't archive over months) and thus in job runtime (which is kept for analysis) not really visible with the naked eye, but looking at the latter using resampling and fft made it really visible. We moved that part of the active monitoring to a later point in the script and selected a different start time in all calling offices...
Yeah, obvious in hindsight. -
@SteveBellovin @mattblaze @hzulla In the late 1990s I worked at a place that had a hundred or so systems all using the same file server. We used a simple wrapper for cron jobs run by all systems to smear out the impact: extract low octet of IP address, sleep that many seconds before starting.
Nowadays I manage about 300 systems, but net and CPU and storage are fast enough that we don't get crowd-load effects. If we did I'd repeat the scheme but sleep N/10 or N/100 seconds.
There's something to be said for changing cron to insert small random delays. I don't think it often (if ever) matters if a job starts a fraction of a second, or even a few seconds, after the exact start of the minute.
@oclsc @SteveBellovin @mattblaze @hzulla nice trick delaying based on the last octet.
Though I might add some other time / date attribute to the mix so lower IP doesn’t always beat higher IP.
But I like the base idea, a lot.
-
UK tv broadcasters had to deliberately stagger their commercial breaks as everyone put the kettle on to make tea during the commercials. And none of that non-British 1kW kettle nonsense, proper British kettles are 3kW!
-
Yes, in the USA, we have 120V through 14g sized wire, which is good enough for about 15 amps. I measure about 1600 watts for devices that claim 1800 watts. Some outlets are rated for 20 amps and those should be wired with 12g sized wire, or property damage is much more likely. For the stoves and clothes dryers, 10g sized wire is used at minimum, rated for 30 amps.
Imagine if all car chargers started at EXACTLY the same hundredth of a second, on the same rising phase of the waveform. You would be able hear the power company turbines ring like a bell from the magnetic hammer.
-
Exactly. Jitter option (-j for users, -J for root) was introduced in cron in FreeBSD 5.3 (late 1993) in order to address system load spikes. The grid effect was probably not evaluated at the time, because CPU/RAM/Network were scarce back then, compared to electricity.
-
@hzulla Cron, of course, dates back to a world in which:
- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational loadSoftware often outlives all sorts of assumptions!
@mattblaze @hzulla Which Cron? I mean there are, for good reasons, many different versions of Cron out there. There are Crons that target specific use cases the original writers didn't consider (like having computers that are turned off or in standby from time to time).
That's the wonderful thing about Free Software. It's not a "product" but instead a chorus of competing ideas and forks which explore new things.
-
RANDOM_DELAY is part of the POSIX standard, so I actually would be very, very surprised, if they don't.
Cronie, anachon, and even systemd supports it.
@shaknais Even if it were available, it seems to randomise by whole minutes, so not potentially easing the problem in this article.
"The RANDOM_DELAY variable allows delaying job startups by random amount of minutes with upper limit specified by the variable."
My current server cron is Vixie, which does not support this feature.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla I have worked on videogames my entire software engineering career. I have thought about the effects of my code on the powergrid. But more in terms of “if my code is inefficient it wastes power, if a million players play the game for so and so hours on their playstation that will amount to so and so many watts-ish”.
-
@datenwolf @hzulla Such an attack is more feasible with a relatively small number of devices if you attack the links between larger electricity networks.
Here's an article from 2017 (German only)
https://www.heise.de/news/Botnetze-koennen-das-Stromnetz-sabotieren-3927886.html?seite=all@newstik @datenwolf @hzulla the original paper is in English
Grid Shock: Coordinated Load-Changing Attacks on Power Grids
https://publications.sba-research.org/publications/201712%20-%20ADabrowski%20-%20Grid%20Shock.pdf
-
@hzulla Cron, of course, dates back to a world in which:
- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational loadSoftware often outlives all sorts of assumptions!
@mattblaze @hzulla I think we need to add "Software often outlives all sorts of assumptions" to one of the "Falsehoods programmers believe about…" awesome lists – perhaps under "Falsehoods programmers believe about time (yes, even astronomers)"…
-
@TimWardCam @hzulla Early adopters are unlikely to bring down the grid by themselves, and the big VPP players are expected to avoid herding also. This is for mainstream timed use, to smear those times.
I haven't run HA, but inserting even a few seconds randomly before bringing on a timed big load will reduce demand slew rate, so I do it routinely, as I have for servers. Random numbers and primes are two of my favourite robustness tools.
@DamonHD @TimWardCam @hzulla My favourite solution is to forget to enable the NTP server.
-
@SteveBellovin @mattblaze @hzulla brb writing a horror novel called “Children of the Cron.”
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
-
@drwho Well, then just turn it on. No need to sleep in cron-jobs.
-
@shaknais Even if it were available, it seems to randomise by whole minutes, so not potentially easing the problem in this article.
"The RANDOM_DELAY variable allows delaying job startups by random amount of minutes with upper limit specified by the variable."
My current server cron is Vixie, which does not support this feature.
Not following. If it randomises by minutes, that flattens the curve, over minutes, and avoids the thundering herd problem.
How would it not?
-
Not following. If it randomises by minutes, that flattens the curve, over minutes, and avoids the thundering herd problem.
How would it not?
@shaknais This article refers to things happening (such as CPU use) at the start of each minute. Merely picking a random minute does not make that particular problem go away. Picking a random second within each minute would. (Better still to pick a random very-fine-grained sub-mains-cycle time to avoid creating a new start-of-second surge problem.)
-
@shaknais This article refers to things happening (such as CPU use) at the start of each minute. Merely picking a random minute does not make that particular problem go away. Picking a random second within each minute would. (Better still to pick a random very-fine-grained sub-mains-cycle time to avoid creating a new start-of-second surge problem.)
@DamonHD Unless everyone picked the same minute, by not seeding their random, you'd be correct.
But, everyone has a different minute. And each task has a different minute.
-
@DamonHD Unless everyone picked the same minute, by not seeding their random, you'd be correct.
But, everyone has a different minute. And each task has a different minute.
@shaknais The problem is that the first few seconds of an average minute have a higher load (see the charts in the Nature paper). Spreading across random minutes helps at other levels, but does not eliminate this feature. When looking at the 1/60Hz spectrum, and eg worrying about mechanical wear on physical turbines, it does not matter much in *which* minutes those early-second surges happen, just that they do happen.
-
@shaknais The problem is that the first few seconds of an average minute have a higher load (see the charts in the Nature paper). Spreading across random minutes helps at other levels, but does not eliminate this feature. When looking at the 1/60Hz spectrum, and eg worrying about mechanical wear on physical turbines, it does not matter much in *which* minutes those early-second surges happen, just that they do happen.
@DamonHD Okay... It doesn't matter which minute, so long as that minute is commonly chosen by _everyone_.
Otherwise you could say it spreading across hours wouldn't flatten the curve - a logical fallacy.
Because of jitter, not everyone is choosing the same moment. Which means there is a lesser load, and a more distributed load, that gracefully increases, and decreases.
If you spread across the 5m interval, it would disappear into the graphs noise.
