This is FASCINATING.
-
@funkylab @benjamineskola My guess it's not cron/systemd daemon waking up on specific times being the problem, but maintainers setting up larger jobs (such as backups) starting at the minute mark. Too many things globally are set to run *something* at xx:xx:00 and this seems to add up to a measurable strain on the grid, thanks to synchronized system clocks. The researchers point out that this strain is not a giant problem yet, but it can be measured globally.
@hzulla @benjamineskola yep, and the discussion here was whether the waking up itself was already part of the problem or the power intense job that you might delay randomly after waking up
-
@davidrylander yeah sure, and it's also the least necessary application of awk to implement something the shell that starts awk can already do itself.
But you wouldn't do that on Debian at all, but just simply use a timer that's randomized.
`systemd-run --on-calendar=your_time_specification --timer-property=RandomizedDelaySec=33 your_command_here`@funkylab Thank you! Must try.
-
@hzulla I believe systemd timers do not start precisely at the minute they are configured, they are executed at a random position within the minute by default (see option
AccuracySec).I have had similar issues with my network before with daily jobs running at midnight, so I have gotten in the habit of setting
RandomizedDelaySecon timers, though it seems likeRandomizedOffsetSecis closer to what I want.@artemist @hzulla AccuracySec and RandomizedDelaySec are for different purposes btw, AccuracySec is for power saving (better to run everything at the same time than keep the cpu awake while one thing is running)
but yeah, if you have a timer that runs once a day, no point in requiring it to run exactly midnight, i went and fixed my backup job system to run at a random time (also might switch to offset instead of delay, looking at it)
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla it's a common problem on cable TV networks where doing an update, or anything that accesses servers, all at the same time brings down the entire network. As well as randomising the point at which you 'wake up' it's a good idea to have a good back-off mechanism for when things fail.
-
sleep "$(awk 'BEGIN{srand(); print int(33*rand());}')"; <rest of cron task>
@DamonHD @GiantGinkgo @hzulla I would just take the entire cronfiles on all your computers and change all 0 minutes to a random value. Thats random enough no?
-
@artemist @hzulla AccuracySec and RandomizedDelaySec are for different purposes btw, AccuracySec is for power saving (better to run everything at the same time than keep the cpu awake while one thing is running)
but yeah, if you have a timer that runs once a day, no point in requiring it to run exactly midnight, i went and fixed my backup job system to run at a random time (also might switch to offset instead of delay, looking at it)
@5225225 @artemist @hzulla it's a bit more nuanced than "AccuracySec is for power saving": Long-term timers are "coarser-grained", so if you need something to happen in 24h 3min 2s (+- 0.2 ms), then you might need to wake up on a coarse (say, 32s-accuracy) timer, and then go back to sleep on a finer timer. If you're like "eh, +-6s is totally OK", then there might be a direct way to do sleep for that amount of time.
-
A anderslund@expressional.social shared this topic
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Something like:
perl -e "sleep(300+int(rand(300)))"
works wonders in a cronjob. Exec it through a script called /usr/local/bin/randomsleep if you want.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla systemd timers actually already have a setting for this, RandomizedDelaySec=! https://www.man7.org/linux/man-pages/man5/systemd.timer.5.html
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Reminds me of Google once self-DDOSing themselves by making every Android phone check for updates exactly every 15minutes without checking if they have the capacity for that spike ...
-
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla They don't actually say how they got synchronized though; how "the start of every minute" somehow lines up across totally different locations.
Presumably the *actual* problem here is that things are connected to the internet and can therefore synchronize their timers precisely?
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
The interesting question is if this randomisation might become necessary in case more and mor datacenters run more and more cron-jobs or other jobs historically timed at fixed hours?
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla in the early Android days, cell network operators were confused by sudden stampedes on a regular schedule. Turns out, all phones were checking for OTAs roughly at the same time. Google didn't notice because to them it was noise and expected.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Crazy how such a small software design detail can have a real impact on the entire power grid. Never thought synchronized tasks could cause this kind of problem.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla systemd timer units have just such a randomising setting but the default is zero so this is in essence a demonstration of the Power of The Default Setting
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
UK tv broadcasters had to deliberately stagger their commercial breaks as everyone put the kettle on to make tea during the commercials. And none of that non-British 1kW kettle nonsense, proper British kettles are 3kW!
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla This is a fairly well known problem. The AWS Well Architected Guide for Sustainability (which I helped write 5 years ago) talks about using cron-jitter to prevent synchronized load spikes in networks and control planes. You also get better performance as jobs aren’t competing for resources. There’s a bigger issue now with AI training runs turning on many GPUs at the same time.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
The Linux Cron doesn't have a -J / -j option?
Sorry, I'm too lazy to look that up, but that's what I'm using on BSD. Not because I care for the power-grid, but It keeps load-spikes at bay (hello, Disk-IO)
-
@josh @hzulla @edavies Ed in particular said words ~20 years ago to the effect of all our NTP-synchronised loads would be hurting the grid...
I introduced NTP to at least one trading floor, where synchronisation is good, but my distributed systems design tools to avoid accidental deadly embraces have long long included randomisation and prime numbers!
-
@DamonHD @GiantGinkgo @hzulla I would just take the entire cronfiles on all your computers and change all 0 minutes to a random value. Thats random enough no?
@hanscees @GiantGinkgo @hzulla No. That's good and helpful and I do a lot of that, but the fact that cron starts everything on the start of *any* minute is part of the issue here.
