This is FASCINATING.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Enterprise scheduling apps like CA AutoSys would kick off thousands if not millions a day, in places like stock brokers. The latency of the event management processor was a big deal. Users all wanting their daily reports kicked off at 08:00:00.000 would cause lags.
Even with multithreading there were not simultaneous bursts of work as they was a logic matrix to work through, unlike cron commands. -
@funkylab @davidrylander @GiantGinkgo @hzulla So I run on a variety of *nixen and shells, and awk works well enough for my purposes. The facility you described arrived long after I started doing some of this stuff, but yes, I am thinking of upgrading.
@DamonHD no, I don't mean systemd-run as what makes awk unnecessary here; the fact that any POSIX shell supports arithmetic expansion `$(( ))` and that basically all korn shell and bourne shell descendants (ksh[93], ash, {Net,Free}BSD bourne sh, OpenBSD [k]sh, bash, zsh, busybox sh…) support `RANDOM` means you can just `sleep $(( RANDOM % 33 ))` here instead of invoking awk to get a random number.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla I don't understand. How does simultaneous power checks dip the frequency (in original paper)? Like I understand that at every minute mark there is some job running, but shouldn't the computation/ power usage be cheap for this?
-
@DamonHD no, I don't mean systemd-run as what makes awk unnecessary here; the fact that any POSIX shell supports arithmetic expansion `$(( ))` and that basically all korn shell and bourne shell descendants (ksh[93], ash, {Net,Free}BSD bourne sh, OpenBSD [k]sh, bash, zsh, busybox sh…) support `RANDOM` means you can just `sleep $(( RANDOM % 33 ))` here instead of invoking awk to get a random number.
@funkylab I don't think I can in sh (actually dash), but certainly couldn't at some point in the past when I first tried. I am an old dog, and once a trick works I tend to keep it as-is!
% sh
$ echo $RANDOM$
So I still can't on my server's sh...
Nor my preferred interactive csh/tcsh:
% echo $RANDOM
RANDOM: Undefined variable.So, none of the shells that I actually use still!
-
@funkylab I don't think I can in sh (actually dash), but certainly couldn't at some point in the past when I first tried. I am an old dog, and once a trick works I tend to keep it as-is!
% sh
$ echo $RANDOM$
So I still can't on my server's sh...
Nor my preferred interactive csh/tcsh:
% echo $RANDOM
RANDOM: Undefined variable.So, none of the shells that I actually use still!
@DamonHD huh, strange, what's your server's `sh` exactly (as in version? does `dash` have an especially restrictive POSIX mode?) Because, it doesn't seem to be in the listing of shells I just gave, or I'm missing something. It's not like I didn't look before I wrote that list
-
@DamonHD huh, strange, what's your server's `sh` exactly (as in version? does `dash` have an especially restrictive POSIX mode?) Because, it doesn't seem to be in the listing of shells I just gave, or I'm missing something. It's not like I didn't look before I wrote that list
@funkylab My server's sh is actually dash, but I have been using *nix systems long enough for sh to have actually being the sh with a 56kB memory footprint!
-
@funkylab My server's sh is actually dash, but I have been using *nix systems long enough for sh to have actually being the sh with a 56kB memory footprint!
@DamonHD yes, but as said, the OG korn shell does this. And that thing is from 1983. I am always a bit interested in computational history, but never in "my beard is greyer than yours, my neck less kempt!" competitions.
-
@funkylab I mean, yeah, if you modified cron to do so. I was assuming that this was adding a random sleep to the start of the cron job itself, which was perhaps a misinterpretation. (I think OpenBSD cron already has this feature, but I’m not sure if it avoids the wake up or only delays the start of the task; and I haven’t seen it in other crons.)
Although of course this might be a premature optimisation, because it hasn’t been shown that there’s a measurable amount of energy being used by cron wakeups in the first place.
@benjamineskola @funkylab @hzulla maybe I'm off the mark here but cron implementations have always honored the RANDOM_DELAY env variable and it's always been recommended to set it to a non-zero value. IIRC systemd timers also honor it but I don't know if it's set by default or not.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla AFAIK the general suggestion for cron jobs that run e.g. 'every hour' is to randomize the second and minute it's set to start at
-
@DamonHD yes, but as said, the OG korn shell does this. And that thing is from 1983. I am always a bit interested in computational history, but never in "my beard is greyer than yours, my neck less kempt!" competitions.
@funkylab As it happens you can see from my profile pic that I am a greybeard. But all I claim here is that none of the shells that I systematically use in dev or prod support(ed) $RANDOM, so that was not a more elegant option open to me. Your experience may well be different and better.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla
Someone(s) in MD has been beating the snot out of one of my plants’ generators. This article may explain why.Lots of extra downtime for generator repairs. (Usually a 20 year rewind we can schedule).
-
@benjamineskola @hzulla It wouldn't create a small spike at the minute mark. Most Linux distributions use SystemD and its timers have this randomization built-in and considered while timer scheduling: https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html#RandomizedDelaySec=
@jak2k @benjamineskola @hzulla I didn't know this was built into systemd, super useful feature!
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla stochastic amortization for the win
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla I always add a random number of seconds to hourly jobs. To avoid load impact. Happy to know that this impact-avoidance stretches further than I thought.
-
@benjamineskola @funkylab @hzulla maybe I'm off the mark here but cron implementations have always honored the RANDOM_DELAY env variable and it's always been recommended to set it to a non-zero value. IIRC systemd timers also honor it but I don't know if it's set by default or not.
@gabrielesvelto @benjamineskola @funkylab @hzulla I don't think any of the cron implementations that I've used have implemented RANDOM_DELAY across many *nixes.
-
At google, the internal cron service has a "@" option in the crontab which effectively hashes the name and uses that to pick a value for the entry so it's stable for a given job but globally distributed.
So instead of `0 0 0 * * *` for running a job daily at midnight, you would do `@ @ 0 * * *` to run it at a randomly chosen but stable time between midnight and 1 am.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Decades ago I read in Sun's Jini spec that services should wait a random short while, before announcing themselves on the LAN to prevent collision, which made me wonder how feasible an electric equivalent would be to prevent a fuse to blow on simultaneous start-up spikes …
-
@funkylab there is a related problem, which is when hourly/daily jobs get run all at the same minute across a large number of machines — updates for example. The issue there is either network traffic or load on the update servers though; and spreading it out doesn’t (and isn’t intended) to reduce the overall load, just to reduce the size of the spikes.
So I guess we also have to consider whether the power spikes are an issue in themselves (if they’re even measurable), or whether we’re more concerned with overall power consumption.
@benjamineskola @funkylab @hzulla
Easily solved with anachron (usual default random delay is 0-45 minutes)
-
@benjamineskola @funkylab @hzulla
Easily solved with anachron (usual default random delay is 0-45 minutes)
@johntimaeus oh yeah, I don’t mean to say it’s a new or unsolved issue, but rather something that’s been a consideration in running scheduled jobs for a long time by now.
-
@hzulla would it actually help? Cron would still wake up at the minute mark in order to sleep again for a few more seconds.
I suppose you’d have two smaller spikes: at the minute, and some seconds later; and the second one would be different for every machine and so average out on a global scale. The first spike would still exist, just smaller, perhaps.
@benjamineskola @hzulla depends. Is cron checking the time the power spike or the actual job itself? I'd expect the on-the-minute bump of cron setting a random delay to be small compared to all the busy work smeared across the rest of the minute.
