This is FASCINATING.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
Ok, so this went surprisingly viral. What I learned from the responses so far:
The main "problem" is the synchronised clock and too many developers using the minute mark for task scheduling default settings.
Having mass-deployed systems doing a thing at the exact same time causing trouble has been known to sysadmins for decades. Solutions readily exist for various task schedulers. However, inexperienced developers and maintainers still default to using the minute mark.
Our industry learns and unlearns this phenomenon again and again.
(I personally certainly have experienced this in my line of work, too, but...)
People may be aware of the problem of synchronised tasks on mass-deployed software for their own distributed system, but I'm mostly surprised that the effect is even visible within the global electricity grid.
So while simple solutions exist since forever, it's unclear how our industry can keep this knowledge alive to avoid future generations doing this little mistake all over again and again.
-
UK tv broadcasters had to deliberately stagger their commercial breaks as everyone put the kettle on to make tea during the commercials. And none of that non-British 1kW kettle nonsense, proper British kettles are 3kW!
-
I learned about cron jobs about 25 years ago. I was told then that it’s good to add a random number of minutes to nightly cron jobs to avoid thundering herds problems. Some tools have this built in:
freebsd-update cron, for example, sleeps a random number of seconds up to an hour to avoid every machine that uses it hammering the server at the same time.I’m not sure when this stopped being taught.
The impact on the electricity grid is a specific case of the same general problem.
The thing that surprises me is that
@dailyis specified to run at an exact time. I had assumed it introduced some jitter to avoid these problems.EDIT: I misremembered the layer of the system responsible for this. On FreeBSD, the main use of
@dailyis to runperiodic daily. This then schedules the periodic tasks, in dependency order. This has aanticongestion_sleeptimesetting and will sleep a random amount up to that amount. The default is 3600, so the daily tasks will start at a random time between midnight and 1am.@david_chisnall @hzulla
Owners of clock shops have known about this problem for centuries, I imagine. -
I learned about cron jobs about 25 years ago. I was told then that it’s good to add a random number of minutes to nightly cron jobs to avoid thundering herds problems. Some tools have this built in:
freebsd-update cron, for example, sleeps a random number of seconds up to an hour to avoid every machine that uses it hammering the server at the same time.I’m not sure when this stopped being taught.
The impact on the electricity grid is a specific case of the same general problem.
The thing that surprises me is that
@dailyis specified to run at an exact time. I had assumed it introduced some jitter to avoid these problems.EDIT: I misremembered the layer of the system responsible for this. On FreeBSD, the main use of
@dailyis to runperiodic daily. This then schedules the periodic tasks, in dependency order. This has aanticongestion_sleeptimesetting and will sleep a random amount up to that amount. The default is 3600, so the daily tasks will start at a random time between midnight and 1am.> I’m not sure when this stopped being taught.
I don't think it ever did stop being taught. I'm at least 10 years after you, I learnt this, younger people than me know this.
-
> I’m not sure when this stopped being taught.
I don't think it ever did stop being taught. I'm at least 10 years after you, I learnt this, younger people than me know this.
I wonder if it’a more that system administration is taught to fewer people. When I was a student, it wasn’t part of the formal curriculum but the computer society gave you hands-on practice administering systems and taught this kind of thing. Maybe 10% of my year group were actively involved. Ten years later it was, apparently, closer to 1%.
-
I wonder if it’a more that system administration is taught to fewer people. When I was a student, it wasn’t part of the formal curriculum but the computer society gave you hands-on practice administering systems and taught this kind of thing. Maybe 10% of my year group were actively involved. Ten years later it was, apparently, closer to 1%.
@david_chisnall @annanannanse I majored in distributed systems while studying CS and it wasn't part of my formal education back in the late 90s. At least not the topic that synchronized clocks can have an effect on the electrical grid.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla lol, yep I've optimised non-critical game system this way.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla According to a friend on another forum, "Octopus put in staggered starts to EV charging sessions to obviate spikes" which seems to cover a large percentage of future demand spikes (in the UK at least).
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Think of all the non-system administrators, regular people, who are now using AI agents to schedule recurring tasks like the ubiquitous “morning brief”. These users aren’t thinking about jitter, and their clankers are doing as asked and delivering the output at for example 7am. This plus the preponderance of network time synchronized OS clocks is going to stack work around common times. Especially business hours.
-
@hzulla Cron, of course, dates back to a world in which:
- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational loadSoftware often outlives all sorts of assumptions!
@mattblaze @hzulla I doubt cron is a relevant part of it.
From the article: "In Ireland, for example, where data centres with large-scale battery storage systems currently account for around a fifth of the country’s electricity demand, the amplitude of the 1-minute signal quadrupled between 2016 and 2021" -
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla that's why most server/industrial computer motherboards have a setting to add a random delay when rebooting...
-
@hzulla would it actually help? Cron would still wake up at the minute mark in order to sleep again for a few more seconds.
I suppose you’d have two smaller spikes: at the minute, and some seconds later; and the second one would be different for every machine and so average out on a global scale. The first spike would still exist, just smaller, perhaps.
@benjamineskola @hzulla the overhead of the cron scheduler is surely going to be only a tiny fraction of the job itself, indistinguishable from idle.
-
@funkylab @benjamineskola @hzulla When I first started with computers we had gotten our hands on some old harddrives that were about 5 cm thick. They had some jumper settings for how many seconds to delay spinup on boot. Since they were pretty heavy the engines would draw too much power if you start two harddrives at the same time and kill the power to the computer. I think the new hdd's we bought when we had money had the same thing for a few years.
My cron jobs are still all running a few minutes off the hour where I can thanks to this experience. Some have randomized wait times between starts.
@eq @funkylab @benjamineskola @hzulla ah yes; remember when (external) hard drives needed to be "spun up" for 15-30 seconds before turning on the machine? My Atari Falcon had a *very* loud Seagate SCSI/ACSI drive that was temperamental like that. Also it's been 16 years since it went in the attic so it's probably a paperweight now.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
Funny though... I have a cron job asking chatGPT for a summary of the 100 hottest news from 32 countries every full hour.
-
@eq @funkylab @benjamineskola @hzulla ah yes; remember when (external) hard drives needed to be "spun up" for 15-30 seconds before turning on the machine? My Atari Falcon had a *very* loud Seagate SCSI/ACSI drive that was temperamental like that. Also it's been 16 years since it went in the attic so it's probably a paperweight now.
@ganxel Yes, I also had a falcon but it had a standard IDE drive since it was rebuilt into a tower by some swedish company I think. It was rather nice until I plugged it in to copy all the files. The powersupply in the house wasn't the best, it gave less than it was supposed to do and that gave too much power to the HDD cable and it started burning.
Beware of low quality third party PC PSU's
-
@psymar @maya_b I really don't think that is why: Canadiens do use electric kettles and they still are safer and work better then the microwave and we are in the same power grid as you.
(You should never boil water in the microwave, there is a small chance if you use a smooth enough glass that it won't boil when it pushes one the BP due to a lack of nucleation sites... until you touch the glass and disturb the water and it suddenly boils all at once)
-
@psymar @maya_b I really don't think that is why: Canadiens do use electric kettles and they still are safer and work better then the microwave and we are in the same power grid as you.
(You should never boil water in the microwave, there is a small chance if you use a smooth enough glass that it won't boil when it pushes one the BP due to a lack of nucleation sites... until you touch the glass and disturb the water and it suddenly boils all at once)
plus electric is so much more convenient that stove top kettles which were the norm when I was growing up.
-
@ganxel Yes, I also had a falcon but it had a standard IDE drive since it was rebuilt into a tower by some swedish company I think. It was rather nice until I plugged it in to copy all the files. The powersupply in the house wasn't the best, it gave less than it was supposed to do and that gave too much power to the HDD cable and it started burning.
Beware of low quality third party PC PSU's
@eq ooh that tower setup sounds neat! Maybe similar to the "Medusa" machines?
-
Ok, so this went surprisingly viral. What I learned from the responses so far:
The main "problem" is the synchronised clock and too many developers using the minute mark for task scheduling default settings.
Having mass-deployed systems doing a thing at the exact same time causing trouble has been known to sysadmins for decades. Solutions readily exist for various task schedulers. However, inexperienced developers and maintainers still default to using the minute mark.
Our industry learns and unlearns this phenomenon again and again.
(I personally certainly have experienced this in my line of work, too, but...)
People may be aware of the problem of synchronised tasks on mass-deployed software for their own distributed system, but I'm mostly surprised that the effect is even visible within the global electricity grid.
So while simple solutions exist since forever, it's unclear how our industry can keep this knowledge alive to avoid future generations doing this little mistake all over again and again.
@hzulla Yeah, lots of people responsible for distributed systems haven’t been taught the ways of thinking about them.
My work team just deployed a scheduled I/O-heavy job (storage snapshots on a non-CoW filesystem) to about a hundred physical servers. The original plan was to push out exactly the same config to all of them. Everybody understood the problem once I pointed it out, but it’s just not something most people consider.
-
Ok, so this went surprisingly viral. What I learned from the responses so far:
The main "problem" is the synchronised clock and too many developers using the minute mark for task scheduling default settings.
Having mass-deployed systems doing a thing at the exact same time causing trouble has been known to sysadmins for decades. Solutions readily exist for various task schedulers. However, inexperienced developers and maintainers still default to using the minute mark.
Our industry learns and unlearns this phenomenon again and again.
(I personally certainly have experienced this in my line of work, too, but...)
People may be aware of the problem of synchronised tasks on mass-deployed software for their own distributed system, but I'm mostly surprised that the effect is even visible within the global electricity grid.
So while simple solutions exist since forever, it's unclear how our industry can keep this knowledge alive to avoid future generations doing this little mistake all over again and again.
@hzulla People also tend to schedule things on the hour/half-hour, or at multiples of five minutes, in part because cron syntax makes that easier.
