This is FASCINATING.
-
@hzulla
Network Time Protocol destroying global infrastructure was not on my bingo card.Your suggested solution makes a lot of sense.
-
@DamonHD @hzulla Nobody has told me to offset my #HomeAssistant automations - so I haven't! And it wouldn't be enforceable anyway, would it?
@TimWardCam @hzulla Early adopters are unlikely to bring down the grid by themselves, and the big VPP players are expected to avoid herding also. This is for mainstream timed use, to smear those times.
I haven't run HA, but inserting even a few seconds randomly before bringing on a timed big load will reduce demand slew rate, so I do it routinely, as I have for servers. Random numbers and primes are two of my favourite robustness tools.
-
@hzulla Similar problem for servers and clusters with on the hour spikes in compute. It was a big issue in my previous job. Especially around 14:00 when most data sources we used had been updated.
Yeah, I know at my last place we had to add deliberate a deliberate couple minutes of jitter to user-scheduled jobs because clients would all choose the same time!
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla wow! Great find!
-
@DamonHD @hzulla Nobody has told me to offset my #HomeAssistant automations - so I haven't! And it wouldn't be enforceable anyway, would it?
@TimWardCam @DamonHD @hzulla the solcast forecasting plugin does this. Too many installs hammering the api at the same time. Not the same as the grid, but still, noticeable
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Props for the very excellent photo of the researchers investigating, halfway down the article
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla * jitter *
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Our power grid is 50 Hz, so I shall make them run every 20 msec.
-
@benjamineskola @hzulla It wouldn't create a small spike at the minute mark. Most Linux distributions use SystemD and its timers have this randomization built-in and considered while timer scheduling: https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html#RandomizedDelaySec=
@jak2k ‘defaults to 0’
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla I've heard something similar about maintenance tasks phones do before. When they're not used at night, there is low activity to reduce battery usage. But then in the morning when you start using your phone again, update checks etc. happen. However because people tend to set their alarms to similar times, the internal schedule adds random padding to when those tasks run to avoid servers getting overloaded with the traffic spike.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla When stopped at traffic lights recently I noticed that all the indicators in the queue of cars behind me were perfectly synchronised. Because of course they were, they were in a precise 1s software-driven loop; nothing uses physical relay/capacitor blinkies any more.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
At google, the internal cron service has a "@" option in the crontab which effectively hashes the name and uses that to pick a value for the entry so it's stable for a given job but globally distributed.
So instead of `0 0 0 * * *` for running a job daily at midnight, you would do `@ @ 0 * * *` to run it at a randomly chosen but stable time between midnight and 1 am.
-
@amethyst We had the additional fun result that since our VMs ground to a halt at 14:00, people moved to scheduled CI jobs instead, so then between 14 and 16 each day, no one could get a PR through because all the runners were queued up for the next 2h.
-
sleep "$(awk 'BEGIN{srand(); print int(33*rand());}')"; <rest of cron task>
@DamonHD @GiantGinkgo @hzulla Does this work when adding cron-job in debian linux? //Linux noob for 25-years.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla being in some of the very early hyperspace AI datacenter space, we were informed by the power company that they were going to disconnect us for drops and spikes that were significant enough to harm the turbines. Its mitigated now, but at that time when a massive jobs stops all gpus at once, then another job comes in right behind it and starts them all again it could literally break the generation
-
@hzulla would it actually help? Cron would still wake up at the minute mark in order to sleep again for a few more seconds.
I suppose you’d have two smaller spikes: at the minute, and some seconds later; and the second one would be different for every machine and so average out on a global scale. The first spike would still exist, just smaller, perhaps.
@benjamineskola @hzulla I think that premise, "cron would wake up on the minute mark" is incorrect: you'd randomize the time it sleeps, in both positive and negative directions Compare how systemd-timers sleep; they already have a well-configurable randomization of wake times, exactly to combat loading a shared resource with a large number is simultaneously triggering machines.
-
@benjamineskola @hzulla I think that premise, "cron would wake up on the minute mark" is incorrect: you'd randomize the time it sleeps, in both positive and negative directions Compare how systemd-timers sleep; they already have a well-configurable randomization of wake times, exactly to combat loading a shared resource with a large number is simultaneously triggering machines.
@funkylab I mean, yeah, if you modified cron to do so. I was assuming that this was adding a random sleep to the start of the cron job itself, which was perhaps a misinterpretation. (I think OpenBSD cron already has this feature, but I’m not sure if it avoids the wake up or only delays the start of the task; and I haven’t seen it in other crons.)
Although of course this might be a premature optimisation, because it hasn’t been shown that there’s a measurable amount of energy being used by cron wakeups in the first place.
-
@DamonHD @GiantGinkgo @hzulla Does this work when adding cron-job in debian linux? //Linux noob for 25-years.
@davidrylander yeah sure, and it's also the least necessary application of awk to implement something the shell that starts awk can already do itself.
But you wouldn't do that on Debian at all, but just simply use a timer that's randomized.
`systemd-run --on-calendar=your_time_specification --timer-property=RandomizedDelaySec=33 your_command_here` -
@funkylab I mean, yeah, if you modified cron to do so. I was assuming that this was adding a random sleep to the start of the cron job itself, which was perhaps a misinterpretation. (I think OpenBSD cron already has this feature, but I’m not sure if it avoids the wake up or only delays the start of the task; and I haven’t seen it in other crons.)
Although of course this might be a premature optimisation, because it hasn’t been shown that there’s a measurable amount of energy being used by cron wakeups in the first place.
@benjamineskola @hzulla agreed. Waking up happens hundreds to thousands of times per second to a modern application processor running an OS like Linux, Windows, *BSD anyways. The extra cost of drawing a pseudorandom or even random number and going back to sleep would be negligible to impossible to measure, even at large scales.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla regarding; https://infosec.exchange/@hzulla/117535147965896440
These things are interesting sir.
https://ethplorer.io/search/aimodel
🟨
