This is FASCINATING.
-
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla This is actually something that I think about!
Maybe not for thr grid exactly, but for anything with more than a couple of instances I want to be nice to services. A ticket at work someone got is because a client isn't staggering something so it absolutely hammers our product.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Wow i've been doing this intuitively for ages without even realizing this. Never schedule exactly on the hour/minute alway stagger/randomize.
-
Ok, so this went surprisingly viral. What I learned from the responses so far:
The main "problem" is the synchronised clock and too many developers using the minute mark for task scheduling default settings.
Having mass-deployed systems doing a thing at the exact same time causing trouble has been known to sysadmins for decades. Solutions readily exist for various task schedulers. However, inexperienced developers and maintainers still default to using the minute mark.
Our industry learns and unlearns this phenomenon again and again.
(I personally certainly have experienced this in my line of work, too, but...)
People may be aware of the problem of synchronised tasks on mass-deployed software for their own distributed system, but I'm mostly surprised that the effect is even visible within the global electricity grid.
So while simple solutions exist since forever, it's unclear how our industry can keep this knowledge alive to avoid future generations doing this little mistake all over again and again.
@hzulla In fairness, devs sticking to the minute mark isn't really something you want to fix (because you can't). Instead you have to recognize what causes it and give devs the tools to avoid it by default. Because there's currently not a good way to express "once a minute, regardless of where in the period" on many platforms
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla as someone who helps operate cloud infrastructure I can truly believe it. Please for th love of god please learn how to jitter
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla @knowprose Apple should do something. We wouldn't even know. They play games and lie about our batteries...
-
@maya_b
Got this from a Danish power plant operator many years ago: They were of course not allowed TV or anything distracting in the control room, so they couldn't watch the big football internationals. They still knew the result as soon as the game ended.If Denmark had won, people would watch the TV a bit longer, enjoying the moment. If Denmark had lost, everybody would turn off, get up and put on the coffee machine right away. The load spike was noticeable.
@hzulla@notsoloud @maya_b @hzulla This is called "TV pickup" in the UK. It is diminishing a bit as there are fewer intense shared moments now...
-
@notsoloud @maya_b @hzulla This is called "TV pickup" in the UK. It is diminishing a bit as there are fewer intense shared moments now...
and also more channels to choose from - not everyone watching the same 3 or 4 things anymore
-
@notsoloud @maya_b @hzulla This is called "TV pickup" in the UK. It is diminishing a bit as there are fewer intense shared moments now...
@DamonHD @notsoloud @maya_b it's also a major plot point in the wonderful animated movie "Flushed Away".
-
@casandro @hzulla Um, if open source or free software or whatever buzzword the kids are calling it these days solves this problem, why wasn’t it solved already? As you pointed out, cron’s code has been re-implemented many times over the years.
Open source is nice and all, but it didn’t anticipate or solve this.
@mattblaze @casandro @hzulla This isn't really a problem that requires changes in the software but rather changes in the way it's used. There is no escaping the necessity of *some* tasks being scheduled at the exact times provided by the various crons. I used to run a fairly big website on meagre hardware and had to dither a lot of scripts to spread CPU load out, this included cron jobs. This also smoothes the power draw.
-
@mattblaze @casandro @hzulla This isn't really a problem that requires changes in the software but rather changes in the way it's used. There is no escaping the necessity of *some* tasks being scheduled at the exact times provided by the various crons. I used to run a fairly big website on meagre hardware and had to dither a lot of scripts to spread CPU load out, this included cron jobs. This also smoothes the power draw.
@synx508 @casandro @hzulla yes, exactly. The software changes required to have cron behave in a way that’s more friendly to the power grid are relatively trivial here. The issue is (apparently) that no one realized (or cared) that synchronized surges in computation can now have a measurable impact at scale (which is an interesting effect).
-
@synx508 @casandro @hzulla yes, exactly. The software changes required to have cron behave in a way that’s more friendly to the power grid are relatively trivial here. The issue is (apparently) that no one realized (or cared) that synchronized surges in computation can now have a measurable impact at scale (which is an interesting effect).
@mattblaze @casandro @hzulla I was aware at the time, this was 20 years ago, I wondered about the effect on the data centre's power distribution back then as we did have metered power as well as data transfer. It wasn't my primary motivation, though. The British anecdote about TV viewers putting the kettle on during half-time of football matches or after a particular soap on TV was quite well known so that scaling effect was, too, but with a different cron engine - television schedules. We didn't have many channels to choose from before the 1980s so the effect was likely more pronounced then.
-
@mattblaze @casandro @hzulla This isn't really a problem that requires changes in the software but rather changes in the way it's used. There is no escaping the necessity of *some* tasks being scheduled at the exact times provided by the various crons. I used to run a fairly big website on meagre hardware and had to dither a lot of scripts to spread CPU load out, this included cron jobs. This also smoothes the power draw.
@synx508 @mattblaze @hzulla Well unfortunately cron only allows one to set the minute, not the second. AFAIK it was always understood that this time was not supposed to be precise. Just adding some randomness probably isn't to hard to do.
-
@drscriptt @SteveBellovin @mattblaze @hzulla It's not about randomizing, really, just spreading. It's OK if several systems hit at the same time, just not the whole head.
@oclsc @SteveBellovin @mattblaze @hzulla
ACK all around.
Reduce the size of the thundering herd to multiple discrete roaming packs of wild servers.
-
@synx508 @mattblaze @hzulla Well unfortunately cron only allows one to set the minute, not the second. AFAIK it was always understood that this time was not supposed to be precise. Just adding some randomness probably isn't to hard to do.
@casandro @mattblaze @hzulla You can do that by wrapping anything called by cron in a "sleep random number of seconds" script, it's trivial and it's not worth adding complexity to cron for in my opinion. Tools should be simple and easily understood.
-
@casandro @mattblaze @hzulla You can do that by wrapping anything called by cron in a "sleep random number of seconds" script, it's trivial and it's not worth adding complexity to cron for in my opinion. Tools should be simple and easily understood.
@synx508 @mattblaze @hzulla Yes, but that would still create the first spike, loading and executing that script.
-
@casandro @mattblaze @hzulla wrapping scripts or modifying scripts isn't what I'd call complex and is "self-declaring" and doesn't change the underlying behaviour of cron, which is something you mostly don't want because a lot of things it does *must* be done at exact times.
If you modified the syntax of crontab so particular lines had the time dither switch that would be potentially complex, if you change defaults so you had to specify "exact" that would be even worse. It's likely not complex in implementation within cron, it's complex in implementation in the real world where this stuff is old and established and making changes like that could be misconstrued etc. -
@hzulla @knowprose Apple should do something. We wouldn't even know. They play games and lie about our batteries...
-
@casandro @mattblaze @hzulla wrapping scripts or modifying scripts isn't what I'd call complex and is "self-declaring" and doesn't change the underlying behaviour of cron, which is something you mostly don't want because a lot of things it does *must* be done at exact times.
If you modified the syntax of crontab so particular lines had the time dither switch that would be potentially complex, if you change defaults so you had to specify "exact" that would be even worse. It's likely not complex in implementation within cron, it's complex in implementation in the real world where this stuff is old and established and making changes like that could be misconstrued etc.@synx508 @mattblaze @hzulla Well most cron don't guarantee precision anyhow, they were not designed for that. It's some daemon running in the background, not a real-time process.
-
@synx508 @mattblaze @hzulla Yes, but that would still create the first spike, loading and executing that script.
@casandro @mattblaze @hzulla I don't think it'd be significant compared to what *could* be unleashed by a cron job starting but I agree, there would be something measurable and consistent. If you really wanted to, the wrapper could be a binary and load from RAM disk, I guess you still have the log to consider, there's no getting away from it. Then even checking the time has a small electrical cost, I suppose you can't remove granularity completely.
