This is FASCINATING.
-
@funkylab @benjamineskola @hzulla When I first started with computers we had gotten our hands on some old harddrives that were about 5 cm thick. They had some jumper settings for how many seconds to delay spinup on boot. Since they were pretty heavy the engines would draw too much power if you start two harddrives at the same time and kill the power to the computer. I think the new hdd's we bought when we had money had the same thing for a few years.
My cron jobs are still all running a few minutes off the hour where I can thanks to this experience. Some have randomized wait times between starts.
@eq @funkylab @benjamineskola @hzulla ah yes; remember when (external) hard drives needed to be "spun up" for 15-30 seconds before turning on the machine? My Atari Falcon had a *very* loud Seagate SCSI/ACSI drive that was temperamental like that. Also it's been 16 years since it went in the attic so it's probably a paperweight now.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
Funny though... I have a cron job asking chatGPT for a summary of the 100 hottest news from 32 countries every full hour.
-
@eq @funkylab @benjamineskola @hzulla ah yes; remember when (external) hard drives needed to be "spun up" for 15-30 seconds before turning on the machine? My Atari Falcon had a *very* loud Seagate SCSI/ACSI drive that was temperamental like that. Also it's been 16 years since it went in the attic so it's probably a paperweight now.
@ganxel Yes, I also had a falcon but it had a standard IDE drive since it was rebuilt into a tower by some swedish company I think. It was rather nice until I plugged it in to copy all the files. The powersupply in the house wasn't the best, it gave less than it was supposed to do and that gave too much power to the HDD cable and it started burning.
Beware of low quality third party PC PSU's
-
@psymar @maya_b I really don't think that is why: Canadiens do use electric kettles and they still are safer and work better then the microwave and we are in the same power grid as you.
(You should never boil water in the microwave, there is a small chance if you use a smooth enough glass that it won't boil when it pushes one the BP due to a lack of nucleation sites... until you touch the glass and disturb the water and it suddenly boils all at once)
-
@psymar @maya_b I really don't think that is why: Canadiens do use electric kettles and they still are safer and work better then the microwave and we are in the same power grid as you.
(You should never boil water in the microwave, there is a small chance if you use a smooth enough glass that it won't boil when it pushes one the BP due to a lack of nucleation sites... until you touch the glass and disturb the water and it suddenly boils all at once)
plus electric is so much more convenient that stove top kettles which were the norm when I was growing up.
-
@ganxel Yes, I also had a falcon but it had a standard IDE drive since it was rebuilt into a tower by some swedish company I think. It was rather nice until I plugged it in to copy all the files. The powersupply in the house wasn't the best, it gave less than it was supposed to do and that gave too much power to the HDD cable and it started burning.
Beware of low quality third party PC PSU's
@eq ooh that tower setup sounds neat! Maybe similar to the "Medusa" machines?
-
Ok, so this went surprisingly viral. What I learned from the responses so far:
The main "problem" is the synchronised clock and too many developers using the minute mark for task scheduling default settings.
Having mass-deployed systems doing a thing at the exact same time causing trouble has been known to sysadmins for decades. Solutions readily exist for various task schedulers. However, inexperienced developers and maintainers still default to using the minute mark.
Our industry learns and unlearns this phenomenon again and again.
(I personally certainly have experienced this in my line of work, too, but...)
People may be aware of the problem of synchronised tasks on mass-deployed software for their own distributed system, but I'm mostly surprised that the effect is even visible within the global electricity grid.
So while simple solutions exist since forever, it's unclear how our industry can keep this knowledge alive to avoid future generations doing this little mistake all over again and again.
@hzulla Yeah, lots of people responsible for distributed systems haven’t been taught the ways of thinking about them.
My work team just deployed a scheduled I/O-heavy job (storage snapshots on a non-CoW filesystem) to about a hundred physical servers. The original plan was to push out exactly the same config to all of them. Everybody understood the problem once I pointed it out, but it’s just not something most people consider.
-
Ok, so this went surprisingly viral. What I learned from the responses so far:
The main "problem" is the synchronised clock and too many developers using the minute mark for task scheduling default settings.
Having mass-deployed systems doing a thing at the exact same time causing trouble has been known to sysadmins for decades. Solutions readily exist for various task schedulers. However, inexperienced developers and maintainers still default to using the minute mark.
Our industry learns and unlearns this phenomenon again and again.
(I personally certainly have experienced this in my line of work, too, but...)
People may be aware of the problem of synchronised tasks on mass-deployed software for their own distributed system, but I'm mostly surprised that the effect is even visible within the global electricity grid.
So while simple solutions exist since forever, it's unclear how our industry can keep this knowledge alive to avoid future generations doing this little mistake all over again and again.
@hzulla People also tend to schedule things on the hour/half-hour, or at multiples of five minutes, in part because cron syntax makes that easier.
-
@oclsc @SteveBellovin @mattblaze @hzulla nice trick delaying based on the last octet.
Though I might add some other time / date attribute to the mix so lower IP doesn’t always beat higher IP.
But I like the base idea, a lot.
@drscriptt @SteveBellovin @mattblaze @hzulla It's not about randomizing, really, just spreading. It's OK if several systems hit at the same time, just not the whole head.
-
Some of us just want hot water. I have an electric kettle which is 15+ yo, a stovetop, several microwave-safe tea pots (yes, they warn about the sudden boil effect, a 'feature' when adding the tea strainer of leaves) and recently sold a cast iron kettle (I no longer have a solid fuel heater, but the buyer does.)
I miss the on-demand thermocouple water tap supplying 99°.
-
Some of us just want hot water. I have an electric kettle which is 15+ yo, a stovetop, several microwave-safe tea pots (yes, they warn about the sudden boil effect, a 'feature' when adding the tea strainer of leaves) and recently sold a cast iron kettle (I no longer have a solid fuel heater, but the buyer does.)
I miss the on-demand thermocouple water tap supplying 99°.
-
Some of us just want hot water. I have an electric kettle which is 15+ yo, a stovetop, several microwave-safe tea pots (yes, they warn about the sudden boil effect, a 'feature' when adding the tea strainer of leaves) and recently sold a cast iron kettle (I no longer have a solid fuel heater, but the buyer does.)
I miss the on-demand thermocouple water tap supplying 99°.
-
-
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting on the global electricity grid.
@hzulla This is actually something that I think about!
Maybe not for thr grid exactly, but for anything with more than a couple of instances I want to be nice to services. A ticket at work someone got is because a client isn't staggering something so it absolutely hammers our product.
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla Wow i've been doing this intuitively for ages without even realizing this. Never schedule exactly on the hour/minute alway stagger/randomize.
-
Ok, so this went surprisingly viral. What I learned from the responses so far:
The main "problem" is the synchronised clock and too many developers using the minute mark for task scheduling default settings.
Having mass-deployed systems doing a thing at the exact same time causing trouble has been known to sysadmins for decades. Solutions readily exist for various task schedulers. However, inexperienced developers and maintainers still default to using the minute mark.
Our industry learns and unlearns this phenomenon again and again.
(I personally certainly have experienced this in my line of work, too, but...)
People may be aware of the problem of synchronised tasks on mass-deployed software for their own distributed system, but I'm mostly surprised that the effect is even visible within the global electricity grid.
So while simple solutions exist since forever, it's unclear how our industry can keep this knowledge alive to avoid future generations doing this little mistake all over again and again.
@hzulla In fairness, devs sticking to the minute mark isn't really something you want to fix (because you can't). Instead you have to recognize what causes it and give devs the tools to avoid it by default. Because there's currently not a good way to express "once a minute, regardless of where in the period" on many platforms
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla as someone who helps operate cloud infrastructure I can truly believe it. Please for th love of god please learn how to jitter
-
This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
@hzulla @knowprose Apple should do something. We wouldn't even know. They play games and lie about our batteries...
-
@maya_b
Got this from a Danish power plant operator many years ago: They were of course not allowed TV or anything distracting in the control room, so they couldn't watch the big football internationals. They still knew the result as soon as the game ended.If Denmark had won, people would watch the TV a bit longer, enjoying the moment. If Denmark had lost, everybody would turn off, get up and put on the coffee machine right away. The load spike was noticeable.
@hzulla@notsoloud @maya_b @hzulla This is called "TV pickup" in the UK. It is diminishing a bit as there are fewer intense shared moments now...
-
@notsoloud @maya_b @hzulla This is called "TV pickup" in the UK. It is diminishing a bit as there are fewer intense shared moments now...
and also more channels to choose from - not everyone watching the same 3 or 4 things anymore
