This is FASCINATING.
-
@hzulla would it actually help? Cron would still wake up at the minute mark in order to sleep again for a few more seconds.
I suppose you’d have two smaller spikes: at the minute, and some seconds later; and the second one would be different for every machine and so average out on a global scale. The first spike would still exist, just smaller, perhaps.
@benjamineskola @hzulla It wouldn't create a small spike at the minute mark. Most Linux distributions use SystemD and its timers have this randomization built-in and considered while timer scheduling: https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html#RandomizedDelaySec=
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla speak for yourself, systemd has had jitter settings for a bunch of years, that's something one notices as soon as measurments are taken.
honorable mention: https://eco.kde.org/blog/
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla I’ve always found it frustrating that the task scheduler defaults this way, knowing that jobs should be spread out… I’d love a randomize function to switch things up in the UI.
-
@hzulla speak for yourself, systemd has had jitter settings for a bunch of years, that's something one notices as soon as measurments are taken.
honorable mention: https://eco.kde.org/blog/
@jane namely RandomizedDelaySec in systemd.timer.
It was probably intended to avoid CPU and network congestion, but it’s interesting that we also have to think about power spikes.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla
Network Time Protocol destroying global infrastructure was not on my bingo card.Your suggested solution makes a lot of sense.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla I already try to randomise my grid interactions by at least a few seconds when driven by cron for this reason!
Eg see Table 4: https://www.academia.edu/2998-3665/3/3/10.20935/AcadEnergy8514
Stick this at the front of your cron jobs, varying the 33 to taste:
sleep "$(awk 'BEGIN{srand(); print int(33*rand());}')";
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla I think that the protocols for charging a car from a timer includes rules for randomising the actual start time.
-
@hzulla One does however wonder about the effect of switching domestic scale consumption/generation on the half hour when (in the UK) the electricity price changes. Obvs you want to switch your battery from fully charging to fully discharging exactly on the half hour because that makes you the most money ... but what if everybody does this?
@TimWardCam @hzulla It's called 'herding' by NESO and (eg) the 10-minute randomisation in car charging start is intended to blunt this...
-
@hzulla I’ve always found it frustrating that the task scheduler defaults this way, knowing that jobs should be spread out… I’d love a randomize function to switch things up in the UI.
sleep "$(awk 'BEGIN{srand(); print int(33*rand());}')"; <rest of cron task>
-
@TimWardCam @hzulla It's called 'herding' by NESO and (eg) the 10-minute randomisation in car charging start is intended to blunt this...
@DamonHD @hzulla Nobody has told me to offset my #HomeAssistant automations - so I haven't! And it wouldn't be enforceable anyway, would it?
-
@hzulla
Network Time Protocol destroying global infrastructure was not on my bingo card.Your suggested solution makes a lot of sense.
-
@DamonHD @hzulla Nobody has told me to offset my #HomeAssistant automations - so I haven't! And it wouldn't be enforceable anyway, would it?
@TimWardCam @hzulla Early adopters are unlikely to bring down the grid by themselves, and the big VPP players are expected to avoid herding also. This is for mainstream timed use, to smear those times.
I haven't run HA, but inserting even a few seconds randomly before bringing on a timed big load will reduce demand slew rate, so I do it routinely, as I have for servers. Random numbers and primes are two of my favourite robustness tools.
-
@hzulla Similar problem for servers and clusters with on the hour spikes in compute. It was a big issue in my previous job. Especially around 14:00 when most data sources we used had been updated.
Yeah, I know at my last place we had to add deliberate a deliberate couple minutes of jitter to user-scheduled jobs because clients would all choose the same time!
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla wow! Great find!
-
@DamonHD @hzulla Nobody has told me to offset my #HomeAssistant automations - so I haven't! And it wouldn't be enforceable anyway, would it?
@TimWardCam @DamonHD @hzulla the solcast forecasting plugin does this. Too many installs hammering the api at the same time. Not the same as the grid, but still, noticeable
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Props for the very excellent photo of the researchers investigating, halfway down the article
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla * jitter *
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Our power grid is 50 Hz, so I shall make them run every 20 msec.
-
@benjamineskola @hzulla It wouldn't create a small spike at the minute mark. Most Linux distributions use SystemD and its timers have this randomization built-in and considered while timer scheduling: https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html#RandomizedDelaySec=
@jak2k ‘defaults to 0’
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla I've heard something similar about maintenance tasks phones do before. When they're not used at night, there is low activity to reduce battery usage. But then in the morning when you start using your phone again, update checks etc. happen. However because people tend to set their alarms to similar times, the internal schedule adds random padding to when those tasks run to avoid servers getting overloaded with the traffic spike.
