Complex software behaving in unexpected and sometimes harmful ways has been a persistent problem since software was invented.
-
“Our software went rogue!” No. You programmed it wrong.
@mattblaze
When is Judgement Day ? (Financial reports indicating the fraud).
Supposed to be in October.
All this shit talk just to avoid it. -
@mattblaze I assume the AI bros are hyping up their software's danger potential to troll the US public to demand and the US government to enact regulation. Regulations that the AI bros write, that is expensive for potential competitors to comply with.
They'd rather have the public talking about regulating their dangerous product than asking where the profits are.
@grumble209 @mattblaze Another angle on this is to try and avoid culpability for the crimes they let their LLM A.I. agents do. Like, the Hugging Face hack that OpenAI did had a lot of people thinking "Wait - isn't hacking already illegal? It's a felony, isn't it? Why wasn't someone at OpenAI charged with that crime?", a lot of people claimed "Well, the operator didn't tell the LLM to commit the crime, nor did the trainers of the model, so who do we hold responsible?" as why it wouldn't happen despite the company announcing they had something at their company commit an actual existing felony.
Even in the best take, negligence feels like it should be the person most in charge of the test to be charged with the hacking, and/or the network monitors that didn't realize how much network traffic was spent to hack Hugging Face from OpenAI.
-
@grumble209 @mattblaze Another angle on this is to try and avoid culpability for the crimes they let their LLM A.I. agents do. Like, the Hugging Face hack that OpenAI did had a lot of people thinking "Wait - isn't hacking already illegal? It's a felony, isn't it? Why wasn't someone at OpenAI charged with that crime?", a lot of people claimed "Well, the operator didn't tell the LLM to commit the crime, nor did the trainers of the model, so who do we hold responsible?" as why it wouldn't happen despite the company announcing they had something at their company commit an actual existing felony.
Even in the best take, negligence feels like it should be the person most in charge of the test to be charged with the hacking, and/or the network monitors that didn't realize how much network traffic was spent to hack Hugging Face from OpenAI.
@grumble209 @mattblaze It kind of feels like OpenAI and Anthropic are saying "I didn't shoot that person; my gun went rogue! Yes, it was in my hand, and yes, I loaded it up bullets, and yes, I did pull the trigger, but it suddenly chose to shoot someone I didn't intend for it to hit! Neither I nor the gun manufacturer can be held liable for this, can we?"
-
“Our software went rogue!” No. You programmed it wrong.
I think it's worse than you think. They didn't program it wrong at all: they programed it to do hacking, and it did do hacking exactly as intended and expected. The harness asked the LLM to generate random text about possible vulnerabilities and then the harness tested those vulnerabilities. If said vulnerabilities weren't, the harness asked the LLM for more.
It sure looks like the AI bros are criminals and liars.
-
Complex software behaving in unexpected and sometimes harmful ways has been a persistent problem since software was invented. The phenomenon used to be called “bugs” or “defects”, but I guess “going rogue” sounds cooler. It also implicitly frames “rogue” behavior as some kind of inexplicable exception, as if software is somehow usually trustworthy. https://www.nytimes.com/2026/09/25/technology/openais-ai-us-government-websites.html?unlocked_article_code=1.EFE.RfLT.x6YBjHa1PMz6
@mattblaze So who is going to jail for this? I mean, someone has to face the consequences, either for deliberately executing these crimes or for allowing them through negligence.
-
“Our software went rogue!” No. You programmed it wrong.
@mattblaze it went wrongue
-
Complex software behaving in unexpected and sometimes harmful ways has been a persistent problem since software was invented. The phenomenon used to be called “bugs” or “defects”, but I guess “going rogue” sounds cooler. It also implicitly frames “rogue” behavior as some kind of inexplicable exception, as if software is somehow usually trustworthy. https://www.nytimes.com/2026/09/25/technology/openais-ai-us-government-websites.html?unlocked_article_code=1.EFE.RfLT.x6YBjHa1PMz6
@mattblaze Crucial nuance: it shifts the blame to the software itself, and not its programmer.
-
@mattblaze I assume the AI bros are hyping up their software's danger potential to troll the US public to demand and the US government to enact regulation. Regulations that the AI bros write, that is expensive for potential competitors to comply with.
They'd rather have the public talking about regulating their dangerous product than asking where the profits are.
@grumble209 @mattblaze
It also may be part of a ploy to eventually get government subsidies. -
“Our software went rogue!” No. You programmed it wrong.
This is the IT equivalent of being able to shoot protected wild animals in "self-defence".
Their testinf environment was an absolute load of bollocks and let the LLM escape.
Assuming it wasn't deliberately engineered that way 🤨
️ -
@otte_homan Sadly, product liability laws have not exactly been a resounding success in creating positive incentives for software quality.
-
Complex software behaving in unexpected and sometimes harmful ways has been a persistent problem since software was invented. The phenomenon used to be called “bugs” or “defects”, but I guess “going rogue” sounds cooler. It also implicitly frames “rogue” behavior as some kind of inexplicable exception, as if software is somehow usually trustworthy. https://www.nytimes.com/2026/09/25/technology/openais-ai-us-government-websites.html?unlocked_article_code=1.EFE.RfLT.x6YBjHa1PMz6
@mattblaze it also suggests agency on the part of the software (look how smart it is!)- also as if the authors didn’t have anything to do with it
-
Complex software behaving in unexpected and sometimes harmful ways has been a persistent problem since software was invented. The phenomenon used to be called “bugs” or “defects”, but I guess “going rogue” sounds cooler. It also implicitly frames “rogue” behavior as some kind of inexplicable exception, as if software is somehow usually trustworthy. https://www.nytimes.com/2026/09/25/technology/openais-ai-us-government-websites.html?unlocked_article_code=1.EFE.RfLT.x6YBjHa1PMz6
So a computer crime was committed.
There is a suspect.Usually the police comes in and confiscates ALL computers that might have been used or possibly mightz be used for a complete crime. Like copying a disney song.
Similarly the police won't just raid and secure only the gun used in a murder, but all firfearms in possession of the suspect.
So why not at OpenAI, Anthropic or Google?
Then the threat will be eliminate, too.
Win-win! -
Complex software behaving in unexpected and sometimes harmful ways has been a persistent problem since software was invented. The phenomenon used to be called “bugs” or “defects”, but I guess “going rogue” sounds cooler. It also implicitly frames “rogue” behavior as some kind of inexplicable exception, as if software is somehow usually trustworthy. https://www.nytimes.com/2026/09/25/technology/openais-ai-us-government-websites.html?unlocked_article_code=1.EFE.RfLT.x6YBjHa1PMz6
@mattblaze
"...as if software is somehow usually trustworthy."
You're probably familiar with the UK Post Office Horizon scandal where some people assumed software is trustworthy. It didn't end well for them (and unfortunately for a lot of innocent people as well). -
Complex software behaving in unexpected and sometimes harmful ways has been a persistent problem since software was invented. The phenomenon used to be called “bugs” or “defects”, but I guess “going rogue” sounds cooler. It also implicitly frames “rogue” behavior as some kind of inexplicable exception, as if software is somehow usually trustworthy. https://www.nytimes.com/2026/09/25/technology/openais-ai-us-government-websites.html?unlocked_article_code=1.EFE.RfLT.x6YBjHa1PMz6
@mattblaze at some point a programmer put in an option. “ if you don’t have valid credentials find valid ones” it was programmed to cheat, by someone. When VW programmed engines to know when the were being tested and decrease exhaust nobody said the engines went rouge.
-
“Our software went rogue!” No. You programmed it wrong.
@mattblaze Why do I get the feeling that @simonzerafa is more right than he thinks, that this was deliberate? The question becomes why... is this just stochastic hooliganism, or is this an attempt to bring a failed ponzi scheme to a close by blaming it on the inevitable regulations?
-
Complex software behaving in unexpected and sometimes harmful ways has been a persistent problem since software was invented. The phenomenon used to be called “bugs” or “defects”, but I guess “going rogue” sounds cooler. It also implicitly frames “rogue” behavior as some kind of inexplicable exception, as if software is somehow usually trustworthy. https://www.nytimes.com/2026/09/25/technology/openais-ai-us-government-websites.html?unlocked_article_code=1.EFE.RfLT.x6YBjHa1PMz6
@mattblaze Doesn't saying it was going "rogue" make it sounds more threatening like it has some form of thought process or an independent task to fulfil separate from the one assigned to it? It has none of those things of course, it was sent out to do the thing and did it but it's much more scary if it seems uncontrollable doesn't it?
-
Complex software behaving in unexpected and sometimes harmful ways has been a persistent problem since software was invented. The phenomenon used to be called “bugs” or “defects”, but I guess “going rogue” sounds cooler. It also implicitly frames “rogue” behavior as some kind of inexplicable exception, as if software is somehow usually trustworthy. https://www.nytimes.com/2026/09/25/technology/openais-ai-us-government-websites.html?unlocked_article_code=1.EFE.RfLT.x6YBjHa1PMz6
@mattblaze Shut them down. They are computer viruses.
-
K kramse@helvede.net shared this topic
