This is FASCINATING.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
At google, the internal cron service has a "@" option in the crontab which effectively hashes the name and uses that to pick a value for the entry so it's stable for a given job but globally distributed.
So instead of `0 0 0 * * *` for running a job daily at midnight, you would do `@ @ 0 * * *` to run it at a randomly chosen but stable time between midnight and 1 am.
-
@amethyst We had the additional fun result that since our VMs ground to a halt at 14:00, people moved to scheduled CI jobs instead, so then between 14 and 16 each day, no one could get a PR through because all the runners were queued up for the next 2h.
-
sleep "$(awk 'BEGIN{srand(); print int(33*rand());}')"; <rest of cron task>
@DamonHD @GiantGinkgo @hzulla Does this work when adding cron-job in debian linux? //Linux noob for 25-years.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla being in some of the very early hyperspace AI datacenter space, we were informed by the power company that they were going to disconnect us for drops and spikes that were significant enough to harm the turbines. Its mitigated now, but at that time when a massive jobs stops all gpus at once, then another job comes in right behind it and starts them all again it could literally break the generation
-
@hzulla would it actually help? Cron would still wake up at the minute mark in order to sleep again for a few more seconds.
I suppose you’d have two smaller spikes: at the minute, and some seconds later; and the second one would be different for every machine and so average out on a global scale. The first spike would still exist, just smaller, perhaps.
@benjamineskola @hzulla I think that premise, "cron would wake up on the minute mark" is incorrect: you'd randomize the time it sleeps, in both positive and negative directions Compare how systemd-timers sleep; they already have a well-configurable randomization of wake times, exactly to combat loading a shared resource with a large number is simultaneously triggering machines.
-
@benjamineskola @hzulla I think that premise, "cron would wake up on the minute mark" is incorrect: you'd randomize the time it sleeps, in both positive and negative directions Compare how systemd-timers sleep; they already have a well-configurable randomization of wake times, exactly to combat loading a shared resource with a large number is simultaneously triggering machines.
@funkylab I mean, yeah, if you modified cron to do so. I was assuming that this was adding a random sleep to the start of the cron job itself, which was perhaps a misinterpretation. (I think OpenBSD cron already has this feature, but I’m not sure if it avoids the wake up or only delays the start of the task; and I haven’t seen it in other crons.)
Although of course this might be a premature optimisation, because it hasn’t been shown that there’s a measurable amount of energy being used by cron wakeups in the first place.
-
@DamonHD @GiantGinkgo @hzulla Does this work when adding cron-job in debian linux? //Linux noob for 25-years.
@davidrylander yeah sure, and it's also the least necessary application of awk to implement something the shell that starts awk can already do itself.
But you wouldn't do that on Debian at all, but just simply use a timer that's randomized.
`systemd-run --on-calendar=your_time_specification --timer-property=RandomizedDelaySec=33 your_command_here` -
@funkylab I mean, yeah, if you modified cron to do so. I was assuming that this was adding a random sleep to the start of the cron job itself, which was perhaps a misinterpretation. (I think OpenBSD cron already has this feature, but I’m not sure if it avoids the wake up or only delays the start of the task; and I haven’t seen it in other crons.)
Although of course this might be a premature optimisation, because it hasn’t been shown that there’s a measurable amount of energy being used by cron wakeups in the first place.
@benjamineskola @hzulla agreed. Waking up happens hundreds to thousands of times per second to a modern application processor running an OS like Linux, Windows, *BSD anyways. The extra cost of drawing a pseudorandom or even random number and going back to sleep would be negligible to impossible to measure, even at large scales.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla regarding; https://infosec.exchange/@hzulla/117535147965896440
These things are interesting sir.
https://ethplorer.io/search/aimodel
🟨
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla I did some measurements on grid frequency disturbances a while ago and you can easily deduce any “nice” multiple of a minute and even time of day just by observing frequency.
-
@benjamineskola @hzulla agreed. Waking up happens hundreds to thousands of times per second to a modern application processor running an OS like Linux, Windows, *BSD anyways. The extra cost of drawing a pseudorandom or even random number and going back to sleep would be negligible to impossible to measure, even at large scales.
@funkylab there is a related problem, which is when hourly/daily jobs get run all at the same minute across a large number of machines — updates for example. The issue there is either network traffic or load on the update servers though; and spreading it out doesn’t (and isn’t intended) to reduce the overall load, just to reduce the size of the spikes.
So I guess we also have to consider whether the power spikes are an issue in themselves (if they’re even measurable), or whether we’re more concerned with overall power consumption.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
Uhh...
I certainly do. For the longest time I wondered what would happen if one deployed a malware that
- synchronized clocks by NTP
- shut down all affected systems at precisely the same time on the evening (so that power plants are ramped down quicker)
- and at the time where normally the evening dip would turn upward suddenly turn on all those systems again (ACPI RTC wakeup) and launched some really CPU & GPU heavy load. -
@benjamineskola @hzulla agreed. Waking up happens hundreds to thousands of times per second to a modern application processor running an OS like Linux, Windows, *BSD anyways. The extra cost of drawing a pseudorandom or even random number and going back to sleep would be negligible to impossible to measure, even at large scales.
@funkylab @benjamineskola My guess it's not cron/systemd daemon waking up on specific times being the problem, but maintainers setting up larger jobs (such as backups) starting at the minute mark. Too many things globally are set to run *something* at xx:xx:00 and this seems to add up to a measurable strain on the grid, thanks to synchronized system clocks. The researchers point out that this strain is not a giant problem yet, but it can be measured globally.
-
@funkylab there is a related problem, which is when hourly/daily jobs get run all at the same minute across a large number of machines — updates for example. The issue there is either network traffic or load on the update servers though; and spreading it out doesn’t (and isn’t intended) to reduce the overall load, just to reduce the size of the spikes.
So I guess we also have to consider whether the power spikes are an issue in themselves (if they’re even measurable), or whether we’re more concerned with overall power consumption.
@benjamineskola @hzulla well they certainly are if whatever the machines are doing is suddenly power intense, and if the machines are a significant consumer in the synchronous grid. (Which my guess would be is something data center operators need to negotiate when getting a grid operator to connect up their DC; they need to buy cable capacity and build transformers for Max load, not just median load)
At a residential PCs scale, wouldn't worry about it much. English tea kettles & halftime breaks. -
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla I believe systemd timers do not start precisely at the minute they are configured, they are executed at a random position within the minute by default (see option
AccuracySec).I have had similar issues with my network before with daily jobs running at midnight, so I have gotten in the habit of setting
RandomizedDelaySecon timers, though it seems likeRandomizedOffsetSecis closer to what I want. -
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla same problem exists for widely used APIs. Eg Octopus Energy manages it's price slots in 30 min increments, so hundreds of thousands of Home Assistant jobs all trigger at the same time every half hour. I believe the developers explicitly add jitter to avoid this (I did, in my app).
-
@funkylab @benjamineskola My guess it's not cron/systemd daemon waking up on specific times being the problem, but maintainers setting up larger jobs (such as backups) starting at the minute mark. Too many things globally are set to run *something* at xx:xx:00 and this seems to add up to a measurable strain on the grid, thanks to synchronized system clocks. The researchers point out that this strain is not a giant problem yet, but it can be measured globally.
@hzulla @benjamineskola yep, and the discussion here was whether the waking up itself was already part of the problem or the power intense job that you might delay randomly after waking up
-
@davidrylander yeah sure, and it's also the least necessary application of awk to implement something the shell that starts awk can already do itself.
But you wouldn't do that on Debian at all, but just simply use a timer that's randomized.
`systemd-run --on-calendar=your_time_specification --timer-property=RandomizedDelaySec=33 your_command_here`@funkylab Thank you! Must try.
-
@hzulla I believe systemd timers do not start precisely at the minute they are configured, they are executed at a random position within the minute by default (see option
AccuracySec).I have had similar issues with my network before with daily jobs running at midnight, so I have gotten in the habit of setting
RandomizedDelaySecon timers, though it seems likeRandomizedOffsetSecis closer to what I want.@artemist @hzulla AccuracySec and RandomizedDelaySec are for different purposes btw, AccuracySec is for power saving (better to run everything at the same time than keep the cpu awake while one thing is running)
but yeah, if you have a timer that runs once a day, no point in requiring it to run exactly midnight, i went and fixed my backup job system to run at a random time (also might switch to offset instead of delay, looking at it)
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla it's a common problem on cable TV networks where doing an update, or anything that accesses servers, all at the same time brings down the entire network. As well as randomising the point at which you 'wake up' it's a good idea to have a good back-off mechanism for when things fail.
