This is FASCINATING.
-
we keep unlearning this.
when i worked at a large cable company doing DNS, we noticed a huge query spike every 15 minutes starting on the hour. the spikes were well above the normal running DNS load.
turned out, the set top boxes checked internet working by every one of them all doing the same query in cron at 0/15/30/45...
i dropped our entired DNS infrastructure peak load requirements by just getting them to add a random sleep interval after 0/15/30/45 to even out the query load...
@paul_ipv6 @hzulla I have a WIP book about computer performance optimization at extreme scale, and this issue/technique was already noted in my own outline for it. I sometimes do professional audits of perf/scale optimization or troubleshooting of client's legacy systems, and this is one of the phenom on my own checklist to look for too
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Oh wow, I now even more love how most of my timers are based on elapsed time (monotonic or realtime). -
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla having been product manager for more than one job scheduling system, this is not surprising at all
more details here http://monkeynoodle.org/2024/01/06/task-scheduling-and-slippage/ -
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Huh, is this why Windows' task scheduler has a "delay for a random amount of time" function?
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Twitter went through the same thing 10+ years ago in its data centers with exactly the same solution.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
Abolish UTC. (I assume that the new rule about leap seconds has already eliminated the NIST's Earth Rotation Service.) If we had free market time, time would be like falafel on the boardwalk. The competing time vendors would be continuously repositioning themselves so that their customers would get better power and more efficient operations than everyone else. And a market in information about what time the different time servers were giving out would suddenly appear. And the different time servers would buy advertising from the people who sold the information about whose time was best, increasing the GDP. And casinos can start making book on which time was going to be when, high-speed traders can start arbitraging futures based on the time differences between different time servers. Let 100 flowers blossom!
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla when I worked at evil mega corp, the easiest way to schedule a job would add a random offset, for network health reasons. Perhaps the corp didn't consider the global power grid, as it wasn't directly within their control and monitoring, but they reached the same conclusion about what details to set.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla if only there would be something Like anacron.
-
Uhh...
I certainly do. For the longest time I wondered what would happen if one deployed a malware that
- synchronized clocks by NTP
- shut down all affected systems at precisely the same time on the evening (so that power plants are ramped down quicker)
- and at the time where normally the evening dip would turn upward suddenly turn on all those systems again (ACPI RTC wakeup) and launched some really CPU & GPU heavy load.@datenwolf @hzulla Such an attack is more feasible with a relatively small number of devices if you attack the links between larger electricity networks.
Here's an article from 2017 (German only)
https://www.heise.de/news/Botnetze-koennen-das-Stromnetz-sabotieren-3927886.html?seite=all -
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla When smartphones were relatively new, they would regularly cause issues on mobile networks, because they all checked for new email or other OOT messages at the same time. IIRC it wasn't event the data volume, it was the synchronized messaging traffic to open Mobile data connections.
I understand that issue has been fixed.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla @maxleibman years ago it was recommended to me that a cron jobs schedule should be set to a prime number so that its distribution would be semi random throughout the day, avoids spikes.
I still follow that advice whenever I setup new jobs.
-
@hzulla I don't understand. How does simultaneous power checks dip the frequency (in original paper)? Like I understand that at every minute mark there is some job running, but shouldn't the computation/ power usage be cheap for this?
@closeted_unionist @hzulla If you add significant load to a large electricity grid, the frequency falls, (until the power plant ramps up).
-
@mattblaze @hzulla Even way back when, I generally offset my cron jobs by a few minutes, just to avoid CPU and RAM (“core storage”) conflicts with other cron jobs.
@SteveBellovin @mattblaze @hzulla In the late 1990s I worked at a place that had a hundred or so systems all using the same file server. We used a simple wrapper for cron jobs run by all systems to smear out the impact: extract low octet of IP address, sleep that many seconds before starting.
Nowadays I manage about 300 systems, but net and CPU and storage are fast enough that we don't get crowd-load effects. If we did I'd repeat the scheme but sleep N/10 or N/100 seconds.
There's something to be said for changing cron to insert small random delays. I don't think it often (if ever) matters if a job starts a fraction of a second, or even a few seconds, after the exact start of the minute.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla since I saw the effects of a "thundering herd", where ten thousands of nodes in a large SaltStack-driven infrastructure all acted at the very same millisecond gave us major headaches, I will always think twice before scheduling anything.
Luckily, many schedulers allow now for mitigating those kind of issues by defining some randomness for distributing the load across a defined timespan. -
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla more for constraints on local compute capacity than the global power grid, but I can't be the only sysadmin who routinely staggers recurring jobs likely updates or backups to kick off at semi-random times like 02:17, 03:43, etc., based on exactly this sort of reasoning…
-
@hzulla new 500-mile email just dropped
-
@jak2k ‘defaults to 0’
-
@benjamineskola @funkylab @hzulla maybe I'm off the mark here but cron implementations have always honored the RANDOM_DELAY env variable and it's always been recommended to set it to a non-zero value. IIRC systemd timers also honor it but I don't know if it's set by default or not.
-
@itgrrl @benjamineskola @jak2k Several have been rejected.
-
@benjamineskola @hzulla It wouldn't create a small spike at the minute mark. Most Linux distributions use SystemD and its timers have this randomization built-in and considered while timer scheduling: https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html#RandomizedDelaySec=
@jak2k @benjamineskola @hzulla Vixie cron supports jitter, but defaults to "none.'
Cronie supports it, and it picks a random jitter value on boot or whenever it restarts. I use it on my boxen, that's how I stumbled across it.
I've also found this technique useful: https://cronradar.com/blog/cron-midnight-load-spreading
