This is FASCINATING.
-
@benjamineskola @funkylab @hzulla maybe I'm off the mark here but cron implementations have always honored the RANDOM_DELAY env variable and it's always been recommended to set it to a non-zero value. IIRC systemd timers also honor it but I don't know if it's set by default or not.
-
@itgrrl @benjamineskola @jak2k Several have been rejected.
-
@benjamineskola @hzulla It wouldn't create a small spike at the minute mark. Most Linux distributions use SystemD and its timers have this randomization built-in and considered while timer scheduling: https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html#RandomizedDelaySec=
@jak2k @benjamineskola @hzulla Vixie cron supports jitter, but defaults to "none.'
Cronie supports it, and it picks a random jitter value on boot or whenever it restarts. I use it on my boxen, that's how I stumbled across it.
I've also found this technique useful: https://cronradar.com/blog/cron-midnight-load-spreading
-
@benjamineskola @funkylab @hzulla
Easily solved with anachron (usual default random delay is 0-45 minutes)
@johntimaeus @benjamineskola @funkylab @hzulla Ooh! Good mention!
-
-
it is for most outlets, but clothes dryer and oven will have special 240V outlets
-
@paul_ipv6 @mattblaze @hzulla My original load projection for Netnews, back when we were designing it in late 1979, as “1–2 messages a day, from 50-100 computers, mostly about Unix system internals”… I was off by far more than two orders of magnitude, of course, which is why I never went in for traffic forecasting.
@SteveBellovin @paul_ipv6 @mattblaze @hzulla Nor I. Early on I was asked to forecast how big a DDoS our link at the office could withstand before we were completely cut off. The next day the first gigabit DDoS attack was detected, easily three zeros above what the office DSL line could handle.
-
@SteveBellovin @mattblaze @hzulla brb writing a horror novel called “Children of the Cron.”
@20002ist @SteveBellovin @mattblaze @hzulla Sequel to _Crop Circles In the Korn Shell_?
-
@hzulla more for constraints on local compute capacity than the global power grid, but I can't be the only sysadmin who routinely staggers recurring jobs likely updates or backups to kick off at semi-random times like 02:17, 03:43, etc., based on exactly this sort of reasoning…
-
@hzulla Cron, of course, dates back to a world in which:
- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational loadSoftware often outlives all sorts of assumptions!
@mattblaze @hzulla i was thinking of this earlier after reading the news. Cron is now harming the grid because of course we didn’t think of that. Haha
-
The Linux Cron doesn't have a -J / -j option?
Sorry, I'm too lazy to look that up, but that's what I'm using on BSD. Not because I care for the power-grid, but It keeps load-spikes at bay (hello, Disk-IO)
-
@gabrielesvelto @benjamineskola @funkylab @hzulla I don't think any of the cron implementations that I've used have implemented RANDOM_DELAY across many *nixes.
RANDOM_DELAY is part of the POSIX standard, so I actually would be very, very surprised, if they don't.
Cronie, anachon, and even systemd supports it.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
Interesting. I've always assumed this and im allergic to :00 times. Probably i absorbed this from time immemorial and forgot it.
It also seems obvious. Not for power spikes, but everything else spikes, cpu etc.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla I asked the question about 4 years ago in the Home Assistant Reddit: https://www.reddit.com/r/homeassistant/s/rgqYwtLDZH
-
UK tv broadcasters had to deliberately stagger their commercial breaks as everyone put the kettle on to make tea during the commercials. And none of that non-British 1kW kettle nonsense, proper British kettles are 3kW!
@maya_b
Got this from a Danish power plant operator many years ago: They were of course not allowed TV or anything distracting in the control room, so they couldn't watch the big football internationals. They still knew the result as soon as the game ended.If Denmark had won, people would watch the TV a bit longer, enjoying the moment. If Denmark had lost, everybody would turn off, get up and put on the coffee machine right away. The load spike was noticeable.
@hzulla -
@amethyst @veronica @hzulla same here - active monitoring would create concurrent spikes in system load (which we don't archive over months) and thus in job runtime (which is kept for analysis) not really visible with the naked eye, but looking at the latter using resampling and fft made it really visible. We moved that part of the active monitoring to a later point in the script and selected a different start time in all calling offices...
Yeah, obvious in hindsight. -
@SteveBellovin @mattblaze @hzulla In the late 1990s I worked at a place that had a hundred or so systems all using the same file server. We used a simple wrapper for cron jobs run by all systems to smear out the impact: extract low octet of IP address, sleep that many seconds before starting.
Nowadays I manage about 300 systems, but net and CPU and storage are fast enough that we don't get crowd-load effects. If we did I'd repeat the scheme but sleep N/10 or N/100 seconds.
There's something to be said for changing cron to insert small random delays. I don't think it often (if ever) matters if a job starts a fraction of a second, or even a few seconds, after the exact start of the minute.
@oclsc @SteveBellovin @mattblaze @hzulla nice trick delaying based on the last octet.
Though I might add some other time / date attribute to the mix so lower IP doesn’t always beat higher IP.
But I like the base idea, a lot.
-
UK tv broadcasters had to deliberately stagger their commercial breaks as everyone put the kettle on to make tea during the commercials. And none of that non-British 1kW kettle nonsense, proper British kettles are 3kW!
-
Yes, in the USA, we have 120V through 14g sized wire, which is good enough for about 15 amps. I measure about 1600 watts for devices that claim 1800 watts. Some outlets are rated for 20 amps and those should be wired with 12g sized wire, or property damage is much more likely. For the stoves and clothes dryers, 10g sized wire is used at minimum, rated for 30 amps.
Imagine if all car chargers started at EXACTLY the same hundredth of a second, on the same rising phase of the waveform. You would be able hear the power company turbines ring like a bell from the magnetic hammer.
-
Exactly. Jitter option (-j for users, -J for root) was introduced in cron in FreeBSD 5.3 (late 1993) in order to address system load spikes. The grid effect was probably not evaluated at the time, because CPU/RAM/Network were scarce back then, compared to electricity.
