This is FASCINATING.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla They don't actually say how they got synchronized though; how "the start of every minute" somehow lines up across totally different locations.
Presumably the *actual* problem here is that things are connected to the internet and can therefore synchronize their timers precisely?
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
The interesting question is if this randomisation might become necessary in case more and mor datacenters run more and more cron-jobs or other jobs historically timed at fixed hours?
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla in the early Android days, cell network operators were confused by sudden stampedes on a regular schedule. Turns out, all phones were checking for OTAs roughly at the same time. Google didn't notice because to them it was noise and expected.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Crazy how such a small software design detail can have a real impact on the entire power grid. Never thought synchronized tasks could cause this kind of problem.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla systemd timer units have just such a randomising setting but the default is zero so this is in essence a demonstration of the Power of The Default Setting
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
UK tv broadcasters had to deliberately stagger their commercial breaks as everyone put the kettle on to make tea during the commercials. And none of that non-British 1kW kettle nonsense, proper British kettles are 3kW!
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla This is a fairly well known problem. The AWS Well Architected Guide for Sustainability (which I helped write 5 years ago) talks about using cron-jitter to prevent synchronized load spikes in networks and control planes. You also get better performance as jobs aren’t competing for resources. There’s a bigger issue now with AI training runs turning on many GPUs at the same time.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
The Linux Cron doesn't have a -J / -j option?
Sorry, I'm too lazy to look that up, but that's what I'm using on BSD. Not because I care for the power-grid, but It keeps load-spikes at bay (hello, Disk-IO)
-
@josh @hzulla @edavies Ed in particular said words ~20 years ago to the effect of all our NTP-synchronised loads would be hurting the grid...
I introduced NTP to at least one trading floor, where synchronisation is good, but my distributed systems design tools to avoid accidental deadly embraces have long long included randomisation and prime numbers!
-
@DamonHD @GiantGinkgo @hzulla I would just take the entire cronfiles on all your computers and change all 0 minutes to a random value. Thats random enough no?
@hanscees @GiantGinkgo @hzulla No. That's good and helpful and I do a lot of that, but the fact that cron starts everything on the start of *any* minute is part of the issue here.
-
@davidrylander yeah sure, and it's also the least necessary application of awk to implement something the shell that starts awk can already do itself.
But you wouldn't do that on Debian at all, but just simply use a timer that's randomized.
`systemd-run --on-calendar=your_time_specification --timer-property=RandomizedDelaySec=33 your_command_here`@funkylab @davidrylander @GiantGinkgo @hzulla So I run on a variety of *nixen and shells, and awk works well enough for my purposes. The facility you described arrived long after I started doing some of this stuff, but yes, I am thinking of upgrading.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Enterprise scheduling apps like CA AutoSys would kick off thousands if not millions a day, in places like stock brokers. The latency of the event management processor was a big deal. Users all wanting their daily reports kicked off at 08:00:00.000 would cause lags.
Even with multithreading there were not simultaneous bursts of work as they was a logic matrix to work through, unlike cron commands. -
@funkylab @davidrylander @GiantGinkgo @hzulla So I run on a variety of *nixen and shells, and awk works well enough for my purposes. The facility you described arrived long after I started doing some of this stuff, but yes, I am thinking of upgrading.
@DamonHD no, I don't mean systemd-run as what makes awk unnecessary here; the fact that any POSIX shell supports arithmetic expansion `$(( ))` and that basically all korn shell and bourne shell descendants (ksh[93], ash, {Net,Free}BSD bourne sh, OpenBSD [k]sh, bash, zsh, busybox sh…) support `RANDOM` means you can just `sleep $(( RANDOM % 33 ))` here instead of invoking awk to get a random number.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla I don't understand. How does simultaneous power checks dip the frequency (in original paper)? Like I understand that at every minute mark there is some job running, but shouldn't the computation/ power usage be cheap for this?
-
@DamonHD no, I don't mean systemd-run as what makes awk unnecessary here; the fact that any POSIX shell supports arithmetic expansion `$(( ))` and that basically all korn shell and bourne shell descendants (ksh[93], ash, {Net,Free}BSD bourne sh, OpenBSD [k]sh, bash, zsh, busybox sh…) support `RANDOM` means you can just `sleep $(( RANDOM % 33 ))` here instead of invoking awk to get a random number.
@funkylab I don't think I can in sh (actually dash), but certainly couldn't at some point in the past when I first tried. I am an old dog, and once a trick works I tend to keep it as-is!
% sh
$ echo $RANDOM$
So I still can't on my server's sh...
Nor my preferred interactive csh/tcsh:
% echo $RANDOM
RANDOM: Undefined variable.So, none of the shells that I actually use still!
-
@funkylab I don't think I can in sh (actually dash), but certainly couldn't at some point in the past when I first tried. I am an old dog, and once a trick works I tend to keep it as-is!
% sh
$ echo $RANDOM$
So I still can't on my server's sh...
Nor my preferred interactive csh/tcsh:
% echo $RANDOM
RANDOM: Undefined variable.So, none of the shells that I actually use still!
@DamonHD huh, strange, what's your server's `sh` exactly (as in version? does `dash` have an especially restrictive POSIX mode?) Because, it doesn't seem to be in the listing of shells I just gave, or I'm missing something. It's not like I didn't look before I wrote that list
-
@DamonHD huh, strange, what's your server's `sh` exactly (as in version? does `dash` have an especially restrictive POSIX mode?) Because, it doesn't seem to be in the listing of shells I just gave, or I'm missing something. It's not like I didn't look before I wrote that list
@funkylab My server's sh is actually dash, but I have been using *nix systems long enough for sh to have actually being the sh with a 56kB memory footprint!
-
@funkylab My server's sh is actually dash, but I have been using *nix systems long enough for sh to have actually being the sh with a 56kB memory footprint!
@DamonHD yes, but as said, the OG korn shell does this. And that thing is from 1983. I am always a bit interested in computational history, but never in "my beard is greyer than yours, my neck less kempt!" competitions.
-
@funkylab I mean, yeah, if you modified cron to do so. I was assuming that this was adding a random sleep to the start of the cron job itself, which was perhaps a misinterpretation. (I think OpenBSD cron already has this feature, but I’m not sure if it avoids the wake up or only delays the start of the task; and I haven’t seen it in other crons.)
Although of course this might be a premature optimisation, because it hasn’t been shown that there’s a measurable amount of energy being used by cron wakeups in the first place.
@benjamineskola @funkylab @hzulla maybe I'm off the mark here but cron implementations have always honored the RANDOM_DELAY env variable and it's always been recommended to set it to a non-zero value. IIRC systemd timers also honor it but I don't know if it's set by default or not.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla AFAIK the general suggestion for cron jobs that run e.g. 'every hour' is to randomize the second and minute it's set to start at
