This is FASCINATING.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla since I saw the effects of a "thundering herd", where ten thousands of nodes in a large SaltStack-driven infrastructure all acted at the very same millisecond gave us major headaches, I will always think twice before scheduling anything.
Luckily, many schedulers allow now for mitigating those kind of issues by defining some randomness for distributing the load across a defined timespan. -
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla more for constraints on local compute capacity than the global power grid, but I can't be the only sysadmin who routinely staggers recurring jobs likely updates or backups to kick off at semi-random times like 02:17, 03:43, etc., based on exactly this sort of reasoning…
-
@hzulla new 500-mile email just dropped
-
@jak2k ‘defaults to 0’
-
@benjamineskola @funkylab @hzulla maybe I'm off the mark here but cron implementations have always honored the RANDOM_DELAY env variable and it's always been recommended to set it to a non-zero value. IIRC systemd timers also honor it but I don't know if it's set by default or not.
-
@itgrrl @benjamineskola @jak2k Several have been rejected.
-
@benjamineskola @hzulla It wouldn't create a small spike at the minute mark. Most Linux distributions use SystemD and its timers have this randomization built-in and considered while timer scheduling: https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html#RandomizedDelaySec=
@jak2k @benjamineskola @hzulla Vixie cron supports jitter, but defaults to "none.'
Cronie supports it, and it picks a random jitter value on boot or whenever it restarts. I use it on my boxen, that's how I stumbled across it.
I've also found this technique useful: https://cronradar.com/blog/cron-midnight-load-spreading
-
@benjamineskola @funkylab @hzulla
Easily solved with anachron (usual default random delay is 0-45 minutes)
@johntimaeus @benjamineskola @funkylab @hzulla Ooh! Good mention!
-
-
it is for most outlets, but clothes dryer and oven will have special 240V outlets
-
@paul_ipv6 @mattblaze @hzulla My original load projection for Netnews, back when we were designing it in late 1979, as “1–2 messages a day, from 50-100 computers, mostly about Unix system internals”… I was off by far more than two orders of magnitude, of course, which is why I never went in for traffic forecasting.
@SteveBellovin @paul_ipv6 @mattblaze @hzulla Nor I. Early on I was asked to forecast how big a DDoS our link at the office could withstand before we were completely cut off. The next day the first gigabit DDoS attack was detected, easily three zeros above what the office DSL line could handle.
-
@SteveBellovin @mattblaze @hzulla brb writing a horror novel called “Children of the Cron.”
@20002ist @SteveBellovin @mattblaze @hzulla Sequel to _Crop Circles In the Korn Shell_?
-
@hzulla more for constraints on local compute capacity than the global power grid, but I can't be the only sysadmin who routinely staggers recurring jobs likely updates or backups to kick off at semi-random times like 02:17, 03:43, etc., based on exactly this sort of reasoning…
-
@hzulla Cron, of course, dates back to a world in which:
- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational loadSoftware often outlives all sorts of assumptions!
@mattblaze @hzulla i was thinking of this earlier after reading the news. Cron is now harming the grid because of course we didn’t think of that. Haha
-
The Linux Cron doesn't have a -J / -j option?
Sorry, I'm too lazy to look that up, but that's what I'm using on BSD. Not because I care for the power-grid, but It keeps load-spikes at bay (hello, Disk-IO)
-
@gabrielesvelto @benjamineskola @funkylab @hzulla I don't think any of the cron implementations that I've used have implemented RANDOM_DELAY across many *nixes.
RANDOM_DELAY is part of the POSIX standard, so I actually would be very, very surprised, if they don't.
Cronie, anachon, and even systemd supports it.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
Interesting. I've always assumed this and im allergic to :00 times. Probably i absorbed this from time immemorial and forgot it.
It also seems obvious. Not for power spikes, but everything else spikes, cpu etc.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla I asked the question about 4 years ago in the Home Assistant Reddit: https://www.reddit.com/r/homeassistant/s/rgqYwtLDZH
-
UK tv broadcasters had to deliberately stagger their commercial breaks as everyone put the kettle on to make tea during the commercials. And none of that non-British 1kW kettle nonsense, proper British kettles are 3kW!
@maya_b
Got this from a Danish power plant operator many years ago: They were of course not allowed TV or anything distracting in the control room, so they couldn't watch the big football internationals. They still knew the result as soon as the game ended.If Denmark had won, people would watch the TV a bit longer, enjoying the moment. If Denmark had lost, everybody would turn off, get up and put on the coffee machine right away. The load spike was noticeable.
@hzulla -
@amethyst @veronica @hzulla same here - active monitoring would create concurrent spikes in system load (which we don't archive over months) and thus in job runtime (which is kept for analysis) not really visible with the naked eye, but looking at the latter using resampling and fft made it really visible. We moved that part of the active monitoring to a later point in the script and selected a different start time in all calling offices...
Yeah, obvious in hindsight.
