No, OpenAI's new magic models did not autonomously hack Huggingface.
-
No, OpenAI's new magic models did not autonomously hack Huggingface.
Per OpenAI's PR blog post (https://openai.com/index/hugging-face-model-evaluation-security-incident/
"This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity."
So they prompted their model running without guardrails to hack some shit. OpenAI told it to do that, the model didn't do shit autonomously (no LLM ever does anything autonomously, it's always prompted).
The whole story is PR. We know that from Anthopic's Mythos: "Look we have this super secret new model and it's so powerful. We are scared ourselves. And you will soon be able to rent it!"
@tante a friend suggests it's co-marketing with Huggingface, if you look at the timeline of announcements
-
@tante a friend suggests it's co-marketing with Huggingface, if you look at the timeline of announcements
@davidgerard 100%.
-
@tante a friend suggests it's co-marketing with Huggingface, if you look at the timeline of announcements
@davidgerard @tante "The last 213 times it was blatant lies and spin, but this time you should be scared!"
-
No, OpenAI's new magic models did not autonomously hack Huggingface.
Per OpenAI's PR blog post (https://openai.com/index/hugging-face-model-evaluation-security-incident/
"This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity."
So they prompted their model running without guardrails to hack some shit. OpenAI told it to do that, the model didn't do shit autonomously (no LLM ever does anything autonomously, it's always prompted).
The whole story is PR. We know that from Anthopic's Mythos: "Look we have this super secret new model and it's so powerful. We are scared ourselves. And you will soon be able to rent it!"
@tante
what the HELL are cyber capabilities.(OTOH, why am I trying to seek meaning in an OpenAI PR release.)
-
@tante
what the HELL are cyber capabilities.(OTOH, why am I trying to seek meaning in an OpenAI PR release.)
@pascoda don't it was mostly generated by an LLM.
-
@tante a friend suggests it's co-marketing with Huggingface, if you look at the timeline of announcements
@davidgerard I can't imagine Huggingface's financials are looking too hot, either.
-
No, OpenAI's new magic models did not autonomously hack Huggingface.
Per OpenAI's PR blog post (https://openai.com/index/hugging-face-model-evaluation-security-incident/
"This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity."
So they prompted their model running without guardrails to hack some shit. OpenAI told it to do that, the model didn't do shit autonomously (no LLM ever does anything autonomously, it's always prompted).
The whole story is PR. We know that from Anthopic's Mythos: "Look we have this super secret new model and it's so powerful. We are scared ourselves. And you will soon be able to rent it!"
@tante
Hadn't heard of "Hugging Face" before. Running out of evil things in Tolkien to name companies & systems for, I guess so here's the face hugger from Alien to be an AI overlord. Lovely. -
@tante
what the HELL are cyber capabilities.(OTOH, why am I trying to seek meaning in an OpenAI PR release.)
-
No, OpenAI's new magic models did not autonomously hack Huggingface.
Per OpenAI's PR blog post (https://openai.com/index/hugging-face-model-evaluation-security-incident/
"This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity."
So they prompted their model running without guardrails to hack some shit. OpenAI told it to do that, the model didn't do shit autonomously (no LLM ever does anything autonomously, it's always prompted).
The whole story is PR. We know that from Anthopic's Mythos: "Look we have this super secret new model and it's so powerful. We are scared ourselves. And you will soon be able to rent it!"
@tante everything is "autonomous" when you put a for loop around it
-
No, OpenAI's new magic models did not autonomously hack Huggingface.
Per OpenAI's PR blog post (https://openai.com/index/hugging-face-model-evaluation-security-incident/
"This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity."
So they prompted their model running without guardrails to hack some shit. OpenAI told it to do that, the model didn't do shit autonomously (no LLM ever does anything autonomously, it's always prompted).
The whole story is PR. We know that from Anthopic's Mythos: "Look we have this super secret new model and it's so powerful. We are scared ourselves. And you will soon be able to rent it!"
@tante I don't feel any particular sympathy for the target in this case, but in general ... when this happens shouldn't the company (OpenAI) be held liable?
If during a weapons evaluation a stray drone accidentally hit a power plant I'm sure the plant owners (and maybe the local residents) would want compensation
-
@tante I don't feel any particular sympathy for the target in this case, but in general ... when this happens shouldn't the company (OpenAI) be held liable?
If during a weapons evaluation a stray drone accidentally hit a power plant I'm sure the plant owners (and maybe the local residents) would want compensation
@michelin well they worked together with Huggingface (on PR and telling them about a bug they may or may not have found). So unless Huggingface sues them who would attack OpenAI?
-
@tante I don't feel any particular sympathy for the target in this case, but in general ... when this happens shouldn't the company (OpenAI) be held liable?
If during a weapons evaluation a stray drone accidentally hit a power plant I'm sure the plant owners (and maybe the local residents) would want compensation
-
@tante a friend suggests it's co-marketing with Huggingface, if you look at the timeline of announcements
@davidgerard @tante The whole story was implausible from the second sentence: Where Huggingface claims that their “AI” detected the attack.
-
@tante but it was impossible to envision that their zero day machine would be able to escape their vibecoded sandbox that ran on a machine connected to the internet. nobody could ever guess that this would happen!
@hagen @tante I don't know, I'd want details on "To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy." and "the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path" before dismissing this...
-
@hagen @tante I don't know, I'd want details on "To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy." and "the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path" before dismissing this...
@SonOfSunTzu @tante that’s what mythos did as well though? you point at something and say go and if you’re willing to pay for compute it goes. it literally did what it was meant to do – and was explicitly told to do. what’s the news here?
-
@pascoda don't it was mostly generated by an LLM.
-
No, OpenAI's new magic models did not autonomously hack Huggingface.
Per OpenAI's PR blog post (https://openai.com/index/hugging-face-model-evaluation-security-incident/
"This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity."
So they prompted their model running without guardrails to hack some shit. OpenAI told it to do that, the model didn't do shit autonomously (no LLM ever does anything autonomously, it's always prompted).
The whole story is PR. We know that from Anthopic's Mythos: "Look we have this super secret new model and it's so powerful. We are scared ourselves. And you will soon be able to rent it!"
@tante How Tagesschau is reporting about it can only be described as refusal to practice journalism. 7 out of 9 paragraphs say "Open AI says" and the other two quote nameless "Experts" and Anthropic's product. And tomorrow I have to discuss with someone again about the possibility of consciousness in token generator software https://www.tagesschau.de/wirtschaft/unternehmen/openai-ki-hackerangriff-100.html
-
@tante
Hadn't heard of "Hugging Face" before. Running out of evil things in Tolkien to name companies & systems for, I guess so here's the face hugger from Alien to be an AI overlord. Lovely.@Gurre @tante No to defend anyone but more to take that concern from you (I would also hate if they start using Alien universe names). The company is named after the hugging face emoji https://en.wikipedia.org/wiki/Hugging_Face
-
@davidgerard @tante The whole story was implausible from the second sentence: Where Huggingface claims that their “AI” detected the attack.
@erlenmayr@chaos.social @davidgerard@circumstances.run @tante@tldr.nettime.org The wild thing about these PR/marketing stunts is that if the "AI" really could do what they're pretending it can do, it'd be doing it all day every day. We'd drown in the announcements. At some point they'd become boring.
This one falls flat on its huggingface right out of the gate.
-
@hagen @tante I don't know, I'd want details on "To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy." and "the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path" before dismissing this...
'How did it get the "stolen credentials"?'
