This is FASCINATING.
-
Exactly. Jitter option (-j for users, -J for root) was introduced in cron in FreeBSD 5.3 (late 1993) in order to address system load spikes. The grid effect was probably not evaluated at the time, because CPU/RAM/Network were scarce back then, compared to electricity.
-
@hzulla Cron, of course, dates back to a world in which:
- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational loadSoftware often outlives all sorts of assumptions!
@mattblaze @hzulla Which Cron? I mean there are, for good reasons, many different versions of Cron out there. There are Crons that target specific use cases the original writers didn't consider (like having computers that are turned off or in standby from time to time).
That's the wonderful thing about Free Software. It's not a "product" but instead a chorus of competing ideas and forks which explore new things.
-
RANDOM_DELAY is part of the POSIX standard, so I actually would be very, very surprised, if they don't.
Cronie, anachon, and even systemd supports it.
@shaknais Even if it were available, it seems to randomise by whole minutes, so not potentially easing the problem in this article.
"The RANDOM_DELAY variable allows delaying job startups by random amount of minutes with upper limit specified by the variable."
My current server cron is Vixie, which does not support this feature.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla I have worked on videogames my entire software engineering career. I have thought about the effects of my code on the powergrid. But more in terms of “if my code is inefficient it wastes power, if a million players play the game for so and so hours on their playstation that will amount to so and so many watts-ish”.
-
@datenwolf @hzulla Such an attack is more feasible with a relatively small number of devices if you attack the links between larger electricity networks.
Here's an article from 2017 (German only)
https://www.heise.de/news/Botnetze-koennen-das-Stromnetz-sabotieren-3927886.html?seite=all@newstik @datenwolf @hzulla the original paper is in English
Grid Shock: Coordinated Load-Changing Attacks on Power Grids
https://publications.sba-research.org/publications/201712%20-%20ADabrowski%20-%20Grid%20Shock.pdf
-
@hzulla Cron, of course, dates back to a world in which:
- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational loadSoftware often outlives all sorts of assumptions!
@mattblaze @hzulla I think we need to add "Software often outlives all sorts of assumptions" to one of the "Falsehoods programmers believe about…" awesome lists – perhaps under "Falsehoods programmers believe about time (yes, even astronomers)"…
-
@TimWardCam @hzulla Early adopters are unlikely to bring down the grid by themselves, and the big VPP players are expected to avoid herding also. This is for mainstream timed use, to smear those times.
I haven't run HA, but inserting even a few seconds randomly before bringing on a timed big load will reduce demand slew rate, so I do it routinely, as I have for servers. Random numbers and primes are two of my favourite robustness tools.
@DamonHD @TimWardCam @hzulla My favourite solution is to forget to enable the NTP server.
-
@SteveBellovin @mattblaze @hzulla brb writing a horror novel called “Children of the Cron.”
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
-
@drwho Well, then just turn it on. No need to sleep in cron-jobs.
-
@shaknais Even if it were available, it seems to randomise by whole minutes, so not potentially easing the problem in this article.
"The RANDOM_DELAY variable allows delaying job startups by random amount of minutes with upper limit specified by the variable."
My current server cron is Vixie, which does not support this feature.
Not following. If it randomises by minutes, that flattens the curve, over minutes, and avoids the thundering herd problem.
How would it not?
-
Not following. If it randomises by minutes, that flattens the curve, over minutes, and avoids the thundering herd problem.
How would it not?
@shaknais This article refers to things happening (such as CPU use) at the start of each minute. Merely picking a random minute does not make that particular problem go away. Picking a random second within each minute would. (Better still to pick a random very-fine-grained sub-mains-cycle time to avoid creating a new start-of-second surge problem.)
-
@shaknais This article refers to things happening (such as CPU use) at the start of each minute. Merely picking a random minute does not make that particular problem go away. Picking a random second within each minute would. (Better still to pick a random very-fine-grained sub-mains-cycle time to avoid creating a new start-of-second surge problem.)
@DamonHD Unless everyone picked the same minute, by not seeding their random, you'd be correct.
But, everyone has a different minute. And each task has a different minute.
-
@DamonHD Unless everyone picked the same minute, by not seeding their random, you'd be correct.
But, everyone has a different minute. And each task has a different minute.
@shaknais The problem is that the first few seconds of an average minute have a higher load (see the charts in the Nature paper). Spreading across random minutes helps at other levels, but does not eliminate this feature. When looking at the 1/60Hz spectrum, and eg worrying about mechanical wear on physical turbines, it does not matter much in *which* minutes those early-second surges happen, just that they do happen.
-
@shaknais The problem is that the first few seconds of an average minute have a higher load (see the charts in the Nature paper). Spreading across random minutes helps at other levels, but does not eliminate this feature. When looking at the 1/60Hz spectrum, and eg worrying about mechanical wear on physical turbines, it does not matter much in *which* minutes those early-second surges happen, just that they do happen.
@DamonHD Okay... It doesn't matter which minute, so long as that minute is commonly chosen by _everyone_.
Otherwise you could say it spreading across hours wouldn't flatten the curve - a logical fallacy.
Because of jitter, not everyone is choosing the same moment. Which means there is a lesser load, and a more distributed load, that gracefully increases, and decreases.
If you spread across the 5m interval, it would disappear into the graphs noise.
-
@mattblaze @hzulla Which Cron? I mean there are, for good reasons, many different versions of Cron out there. There are Crons that target specific use cases the original writers didn't consider (like having computers that are turned off or in standby from time to time).
That's the wonderful thing about Free Software. It's not a "product" but instead a chorus of competing ideas and forks which explore new things.
@casandro @hzulla Um, if open source or free software or whatever buzzword the kids are calling it these days solves this problem, why wasn’t it solved already? As you pointed out, cron’s code has been re-implemented many times over the years.
Open source is nice and all, but it didn’t anticipate or solve this.
-
@DamonHD Okay... It doesn't matter which minute, so long as that minute is commonly chosen by _everyone_.
Otherwise you could say it spreading across hours wouldn't flatten the curve - a logical fallacy.
Because of jitter, not everyone is choosing the same moment. Which means there is a lesser load, and a more distributed load, that gracefully increases, and decreases.
If you spread across the 5m interval, it would disappear into the graphs noise.
@shaknais Synchronised extra work in the start of ANY minute, as fired off by EVERY cron on the planet, will add to *this* problem. Cron (as I have ever used) does not add jitter to its wakeup on the minute, whichever minute that is. (Picking random minutes does reduce *other* problems.)
To be clear: 1/60Hz spikes are independent of 1/3600Hz spikes etc: they should both be dealt with, but have to dealt with separately.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
I learned about cron jobs about 25 years ago. I was told then that it’s good to add a random number of minutes to nightly cron jobs to avoid thundering herds problems. Some tools have this built in:
freebsd-update cron, for example, sleeps a random number of seconds up to an hour to avoid every machine that uses it hammering the server at the same time.I’m not sure when this stopped being taught.
The impact on the electricity grid is a specific case of the same general problem.
The thing that surprises me is that
@dailyis specified to run at an exact time. I had assumed it introduced some jitter to avoid these problems.EDIT: I misremembered the layer of the system responsible for this. On FreeBSD, the main use of
@dailyis to runperiodic daily. This then schedules the periodic tasks, in dependency order. This has aanticongestion_sleeptimesetting and will sleep a random amount up to that amount. The default is 3600, so the daily tasks will start at a random time between midnight and 1am. -
@shaknais Synchronised extra work in the start of ANY minute, as fired off by EVERY cron on the planet, will add to *this* problem. Cron (as I have ever used) does not add jitter to its wakeup on the minute, whichever minute that is. (Picking random minutes does reduce *other* problems.)
To be clear: 1/60Hz spikes are independent of 1/3600Hz spikes etc: they should both be dealt with, but have to dealt with separately.
@DamonHD How would spreading the start time of processes, across 60 new units, instead of 1, suddenly increase load?
How do you get _more_ from _less_? You're distributing.
If what you're saying is true, then PFR wouldn't be working in Australia, because that only spreads across 4-seconds for the 1/50Hz. But it is working.
