This is FASCINATING.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla
Someone(s) in MD has been beating the snot out of one of my plants’ generators. This article may explain why.Lots of extra downtime for generator repairs. (Usually a 20 year rewind we can schedule).
-
@benjamineskola @hzulla It wouldn't create a small spike at the minute mark. Most Linux distributions use SystemD and its timers have this randomization built-in and considered while timer scheduling: https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html#RandomizedDelaySec=
@jak2k @benjamineskola @hzulla I didn't know this was built into systemd, super useful feature!
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla stochastic amortization for the win
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla I always add a random number of seconds to hourly jobs. To avoid load impact. Happy to know that this impact-avoidance stretches further than I thought.
-
@benjamineskola @funkylab @hzulla maybe I'm off the mark here but cron implementations have always honored the RANDOM_DELAY env variable and it's always been recommended to set it to a non-zero value. IIRC systemd timers also honor it but I don't know if it's set by default or not.
@gabrielesvelto @benjamineskola @funkylab @hzulla I don't think any of the cron implementations that I've used have implemented RANDOM_DELAY across many *nixes.
-
At google, the internal cron service has a "@" option in the crontab which effectively hashes the name and uses that to pick a value for the entry so it's stable for a given job but globally distributed.
So instead of `0 0 0 * * *` for running a job daily at midnight, you would do `@ @ 0 * * *` to run it at a randomly chosen but stable time between midnight and 1 am.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Decades ago I read in Sun's Jini spec that services should wait a random short while, before announcing themselves on the LAN to prevent collision, which made me wonder how feasible an electric equivalent would be to prevent a fuse to blow on simultaneous start-up spikes …
-
@funkylab there is a related problem, which is when hourly/daily jobs get run all at the same minute across a large number of machines — updates for example. The issue there is either network traffic or load on the update servers though; and spreading it out doesn’t (and isn’t intended) to reduce the overall load, just to reduce the size of the spikes.
So I guess we also have to consider whether the power spikes are an issue in themselves (if they’re even measurable), or whether we’re more concerned with overall power consumption.
@benjamineskola @funkylab @hzulla
Easily solved with anachron (usual default random delay is 0-45 minutes)
-
@benjamineskola @funkylab @hzulla
Easily solved with anachron (usual default random delay is 0-45 minutes)
@johntimaeus oh yeah, I don’t mean to say it’s a new or unsolved issue, but rather something that’s been a consideration in running scheduled jobs for a long time by now.
-
@hzulla would it actually help? Cron would still wake up at the minute mark in order to sleep again for a few more seconds.
I suppose you’d have two smaller spikes: at the minute, and some seconds later; and the second one would be different for every machine and so average out on a global scale. The first spike would still exist, just smaller, perhaps.
@benjamineskola @hzulla depends. Is cron checking the time the power spike or the actual job itself? I'd expect the on-the-minute bump of cron setting a random delay to be small compared to all the busy work smeared across the rest of the minute.
-
I might have the order wrong but I'm pretty sure I put @ for seconds and minutes and 0 for hours which would start at a random but stable minute and second within the given hour
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla i can add to this story
We once had a service that was integrated into every Windows Mobile that also stupidly checked in our service on a fixed clock. That wasn't a problem as long as Windows Mobile didn't had NTP. But then suddenly Microsoft added that feature and after a short time all phones suddenly had accurate time and DDOSed our service on the fixed interval..... -
@hzulla being in some of the very early hyperspace AI datacenter space, we were informed by the power company that they were going to disconnect us for drops and spikes that were significant enough to harm the turbines. Its mitigated now, but at that time when a massive jobs stops all gpus at once, then another job comes in right behind it and starts them all again it could literally break the generation
@hzulla I know i typed hyerscale, but the auto correct is too funny to edit
-
-
Oh I must be misremembering that part then. It's been a while since I had to write a crontab and I always have to consult the comment at the top and then forget everything once I exit.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
we keep unlearning this.
when i worked at a large cable company doing DNS, we noticed a huge query spike every 15 minutes starting on the hour. the spikes were well above the normal running DNS load.
turned out, the set top boxes checked internet working by every one of them all doing the same query in cron at 0/15/30/45...
i dropped our entired DNS infrastructure peak load requirements by just getting them to add a random sleep interval after 0/15/30/45 to even out the query load...
-
-
@hzulla systemd timer units have just such a randomising setting but the default is zero so this is in essence a demonstration of the Power of The Default Setting
@msh @hzulla I think the default is one minute of potential randomization. https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html?hl=en-US
-
@benjamineskola @hzulla well they certainly are if whatever the machines are doing is suddenly power intense, and if the machines are a significant consumer in the synchronous grid. (Which my guess would be is something data center operators need to negotiate when getting a grid operator to connect up their DC; they need to buy cable capacity and build transformers for Max load, not just median load)
At a residential PCs scale, wouldn't worry about it much. English tea kettles & halftime breaks.@funkylab @benjamineskola @hzulla When I first started with computers we had gotten our hands on some old harddrives that were about 5 cm thick. They had some jumper settings for how many seconds to delay spinup on boot. Since they were pretty heavy the engines would draw too much power if you start two harddrives at the same time and kill the power to the computer. I think the new hdd's we bought when we had money had the same thing for a few years.
My cron jobs are still all running a few minutes off the hour where I can thanks to this experience. Some have randomized wait times between starts.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla Cron, of course, dates back to a world in which:
- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational loadSoftware often outlives all sorts of assumptions!
