This is FASCINATING.
-
-
Oh I must be misremembering that part then. It's been a while since I had to write a crontab and I always have to consult the comment at the top and then forget everything once I exit.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
we keep unlearning this.
when i worked at a large cable company doing DNS, we noticed a huge query spike every 15 minutes starting on the hour. the spikes were well above the normal running DNS load.
turned out, the set top boxes checked internet working by every one of them all doing the same query in cron at 0/15/30/45...
i dropped our entired DNS infrastructure peak load requirements by just getting them to add a random sleep interval after 0/15/30/45 to even out the query load...
-
-
@hzulla systemd timer units have just such a randomising setting but the default is zero so this is in essence a demonstration of the Power of The Default Setting
@msh @hzulla I think the default is one minute of potential randomization. https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html?hl=en-US
-
@benjamineskola @hzulla well they certainly are if whatever the machines are doing is suddenly power intense, and if the machines are a significant consumer in the synchronous grid. (Which my guess would be is something data center operators need to negotiate when getting a grid operator to connect up their DC; they need to buy cable capacity and build transformers for Max load, not just median load)
At a residential PCs scale, wouldn't worry about it much. English tea kettles & halftime breaks.@funkylab @benjamineskola @hzulla When I first started with computers we had gotten our hands on some old harddrives that were about 5 cm thick. They had some jumper settings for how many seconds to delay spinup on boot. Since they were pretty heavy the engines would draw too much power if you start two harddrives at the same time and kill the power to the computer. I think the new hdd's we bought when we had money had the same thing for a few years.
My cron jobs are still all running a few minutes off the hour where I can thanks to this experience. Some have randomized wait times between starts.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla Cron, of course, dates back to a world in which:
- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational loadSoftware often outlives all sorts of assumptions!
-
13A on 240V is doable in most of Europe, but generally all the kettles I've seen outside the UK are only 1kw - or a hairdryer or microwave's worth of power.
-
it is for most outlets, but clothes dryer and oven will have special 240V outlets
-
not without an insane plug adapter!
-
@hzulla Cron, of course, dates back to a world in which:
- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational loadSoftware often outlives all sorts of assumptions!
yup. there is a point at which "expected" or "normal" traffic is a self DDoS if you do enough of it at once.
i always found it useful when planning capacity to take the "expected" and ask "what happens if i add two zeros to that amount".
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla I think there was a talk about this kind of thing a few years ago: https://sysadmin.miniconf.org/2020/lca2020-craig_miskell-tyranny_of_the_clock.pdf
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla The bit that surprises me is that the effect was turned up be researchers looking at grid-scale rather than coming up in the observations of one or more of the large-scale server operators. Datacenter power delivery and backup systems aren't cheap, servers have had pretty decent onboard power metering for a while; and there have certainly been setups with thousands to tens of thousands of homogenous builds well before the current 'AI' craze.
-
@hzulla Cron, of course, dates back to a world in which:
- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational loadSoftware often outlives all sorts of assumptions!
@mattblaze @hzulla Even way back when, I generally offset my cron jobs by a few minutes, just to avoid CPU and RAM (“core storage”) conflicts with other cron jobs.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla we have a big scheduling system and that adds "jitter" to all the jobs.
-
@mattblaze @hzulla Even way back when, I generally offset my cron jobs by a few minutes, just to avoid CPU and RAM (“core storage”) conflicts with other cron jobs.
@SteveBellovin @hzulla Yeah, but back then, you mostly only had to worry about conflicts on the same machine.
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla@infosec.exchange I recall a toolchain maintainer mentioning environmental concerns as one reason he insists on shipping binary packages over source packages (as those would have a building cost on every user of the toolchain) in a conversation with me.
-
yup. there is a point at which "expected" or "normal" traffic is a self DDoS if you do enough of it at once.
i always found it useful when planning capacity to take the "expected" and ask "what happens if i add two zeros to that amount".
@paul_ipv6 @mattblaze @hzulla My original load projection for Netnews, back when we were designing it in late 1979, as “1–2 messages a day, from 50-100 computers, mostly about Unix system internals”… I was off by far more than two orders of magnitude, of course, which is why I never went in for traffic forecasting.
-
@paul_ipv6 @mattblaze @hzulla My original load projection for Netnews, back when we were designing it in late 1979, as “1–2 messages a day, from 50-100 computers, mostly about Unix system internals”… I was off by far more than two orders of magnitude, of course, which is why I never went in for traffic forecasting.
@SteveBellovin @mattblaze @hzulla
i have memories of hard coding a max number of newsgroups of 1024? another vast under-estimation.
when an ISP i worked at hit 1 trillion recursive DNS queries a day, i was appalled.
-
@SteveBellovin @hzulla Yeah, but back then, you mostly only had to worry about conflicts on the same machine.
@mattblaze @SteveBellovin @hzulla we, Windows, see measurable megawatt changes when we do good or bad on perf across the datacenters. It's fascinating to get a report back saying we reduced a megawatt of usage by changing a poll from say one minute to five. Also update reboots. 🥴
