This is FASCINATING.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla The bit that surprises me is that the effect was turned up be researchers looking at grid-scale rather than coming up in the observations of one or more of the large-scale server operators. Datacenter power delivery and backup systems aren't cheap, servers have had pretty decent onboard power metering for a while; and there have certainly been setups with thousands to tens of thousands of homogenous builds well before the current 'AI' craze.
-
@hzulla Cron, of course, dates back to a world in which:
- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational loadSoftware often outlives all sorts of assumptions!
@mattblaze @hzulla Even way back when, I generally offset my cron jobs by a few minutes, just to avoid CPU and RAM (“core storage”) conflicts with other cron jobs.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla we have a big scheduling system and that adds "jitter" to all the jobs.
-
@mattblaze @hzulla Even way back when, I generally offset my cron jobs by a few minutes, just to avoid CPU and RAM (“core storage”) conflicts with other cron jobs.
@SteveBellovin @hzulla Yeah, but back then, you mostly only had to worry about conflicts on the same machine.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla@infosec.exchange I recall a toolchain maintainer mentioning environmental concerns as one reason he insists on shipping binary packages over source packages (as those would have a building cost on every user of the toolchain) in a conversation with me.
-
yup. there is a point at which "expected" or "normal" traffic is a self DDoS if you do enough of it at once.
i always found it useful when planning capacity to take the "expected" and ask "what happens if i add two zeros to that amount".
@paul_ipv6 @mattblaze @hzulla My original load projection for Netnews, back when we were designing it in late 1979, as “1–2 messages a day, from 50-100 computers, mostly about Unix system internals”… I was off by far more than two orders of magnitude, of course, which is why I never went in for traffic forecasting.
-
@paul_ipv6 @mattblaze @hzulla My original load projection for Netnews, back when we were designing it in late 1979, as “1–2 messages a day, from 50-100 computers, mostly about Unix system internals”… I was off by far more than two orders of magnitude, of course, which is why I never went in for traffic forecasting.
@SteveBellovin @mattblaze @hzulla
i have memories of hard coding a max number of newsgroups of 1024? another vast under-estimation.
when an ISP i worked at hit 1 trillion recursive DNS queries a day, i was appalled.
-
@SteveBellovin @hzulla Yeah, but back then, you mostly only had to worry about conflicts on the same machine.
@mattblaze @SteveBellovin @hzulla we, Windows, see measurable megawatt changes when we do good or bad on perf across the datacenters. It's fascinating to get a report back saying we reduced a megawatt of usage by changing a poll from say one minute to five. Also update reboots. 🥴
-
@mattblaze @hzulla Even way back when, I generally offset my cron jobs by a few minutes, just to avoid CPU and RAM (“core storage”) conflicts with other cron jobs.
@SteveBellovin @mattblaze @hzulla vixie cron has random and offset features, which can help some ...
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
Vixie cron has had -j (for jitter) for helpful offsets for a very long time now (e.g. https://man.freebsd.org/cgi/man.cgi?query=cron&sektion=8&apropos=0&manpath=FreeBSD+11.0-RELEASE+and+Ports)
Todd Miller also introduced some similar functionality to OpenBSD's cron in 2023 (e.g. https://undeadly.org/cgi?action=article;sid=20230507122935)
Such things were typically inspired to avoid "hammering" (a different sort of spike nomenclature typically related to bandwidth and computational resource utilization) at specific intervals, yet logically they can also potentially alleviate power grid knock on effects as well.
I'm less certain about Linux distros' cron functionality providing similar things? Though maybe via packages a different cron can be installed with the requisite parameters for ameliorative adjustments if whatever distros' own cron is deficient in such realms?
-
@mattblaze @hzulla Even way back when, I generally offset my cron jobs by a few minutes, just to avoid CPU and RAM (“core storage”) conflicts with other cron jobs.
@SteveBellovin @mattblaze @hzulla brb writing a horror novel called “Children of the Cron.”
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla adding another data point here; when I worked in satellite set top box land we used a hash of the device MAC address to offset the viewing card callback, precisely to avoid the load spikes on the head end infrastructure.. multiple million STBs connected to phone lines can create quite the attack vector 🤪.
Making it predictable on the same box helps the support desk to know when each customers device should update... no need for randomness, just distribution.
-
we keep unlearning this.
when i worked at a large cable company doing DNS, we noticed a huge query spike every 15 minutes starting on the hour. the spikes were well above the normal running DNS load.
turned out, the set top boxes checked internet working by every one of them all doing the same query in cron at 0/15/30/45...
i dropped our entired DNS infrastructure peak load requirements by just getting them to add a random sleep interval after 0/15/30/45 to even out the query load...
@paul_ipv6 @hzulla I have a WIP book about computer performance optimization at extreme scale, and this issue/technique was already noted in my own outline for it. I sometimes do professional audits of perf/scale optimization or troubleshooting of client's legacy systems, and this is one of the phenom on my own checklist to look for too
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Oh wow, I now even more love how most of my timers are based on elapsed time (monotonic or realtime). -
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla having been product manager for more than one job scheduling system, this is not surprising at all
more details here http://monkeynoodle.org/2024/01/06/task-scheduling-and-slippage/ -
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Huh, is this why Windows' task scheduler has a "delay for a random amount of time" function?
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Twitter went through the same thing 10+ years ago in its data centers with exactly the same solution.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
Abolish UTC. (I assume that the new rule about leap seconds has already eliminated the NIST's Earth Rotation Service.) If we had free market time, time would be like falafel on the boardwalk. The competing time vendors would be continuously repositioning themselves so that their customers would get better power and more efficient operations than everyone else. And a market in information about what time the different time servers were giving out would suddenly appear. And the different time servers would buy advertising from the people who sold the information about whose time was best, increasing the GDP. And casinos can start making book on which time was going to be when, high-speed traders can start arbitraging futures based on the time differences between different time servers. Let 100 flowers blossom!
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla when I worked at evil mega corp, the easiest way to schedule a job would add a random offset, for network health reasons. Perhaps the corp didn't consider the global power grid, as it wasn't directly within their control and monitoring, but they reached the same conclusion about what details to set.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla if only there would be something Like anacron.
