Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
-
@devsimsek wow, almost as if this was a problem known as overtraining for well over 30 years
@SRAZKVT Exactly.
-
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about it
https://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
@devsimsek Nobody ever claimed that llms get better by being trained on their own synthetic data. This blog post is very misleading.
The idea of self-improvement and singularity is that llms write improved versions of their own codebase and perform the research and experiments for coming up with better models themselves.
The idea of singularity is interesting but also full of hidden assumptions. I'm always confused when people act like singularity would exist. It's just science fiction. -
@dpiponi @devsimsek I find the paper interesting but I would like to understand the exact
premises. "AI" is not equal to gen AI or LLMs, it probably makes little sense to sell it as a general statement about "AI"@Quantensalat @dpiponi That's what I hate about these companies.
-
@devsimsek So... let me get this straight. Autocoprophagic #RSI •doesn't• lead to #AGI? Say it ain't so!
#AI@ghostinthenet Yep
-
@devsimsek also see https://berryvilleiml.com/2026/01/10/recursive-pollution-and-model-collapse-are-not-the-same/
This is part of a long running #ML research thread with big #MLsec impact
@noplasticshower Thanks, ill look into it
-
@anne_twain YEP, most of the people whom commented assumes they develop every iteration with fresh data; that comes from internet ...
-
@Quantensalat @dpiponi That's what I hate about these companies.
@devsimsek @dpiponi that they act like AI=LLMs?
-
@anne_twain @devsimsek
"That's like a high school history class having their own essays as research material." - a memorable phrase. -
@devsimsek @dpiponi that they act like AI=LLMs?
@Quantensalat @dpiponi yes. I did used the same tactic while naming my post as satire. its annoying....
-
@devsimsek @dpiponi that they act like AI=LLMs?
@Quantensalat @devsimsek There's a setup around equations (9) and (10) where the distribution used for training the next generation is a linear combination of the distribution your current generation generates and external data. As the amount of external data goes to zero, you expect model collapse. This is hardly surprising. I don't know anyone who expects you can just keep training based on previous results and expect something radically new to happen. (Though something *useful* can happen - eg. you may improve performance this way. See "rectification" in flow-matching.)
Note that this doesn't rule out all forms of self-training - just one kind. As a concrete example, an LLM trained to generate code can learn from the output of the generated code. Such output is, in some sense, exogenous.
-
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about ithttps://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
> Human-generated data is irreplaceable. The “internet is running out of training data” problem just got mathematically formalized.
Yeah I think the AI con mob has realized this already (but of course not saying the quiet part out loud). With Satya whining about people calling it slop and the AI industry trying to force it down everyone's throats no matter the cost (e.g. Copilot) I think they realize that there is only so much internet and historical content they can use to train their models - now they want *you* to help train it for them. Prompt Claude to spit out some code, ask Copilot for a PR review, and _interact_ with it, pointing out where it was stupid, confirming when it did a good job, by virtue of interacting with an AI model you are improving it with this exact, essential human input.
-
@Quantensalat @devsimsek There's a setup around equations (9) and (10) where the distribution used for training the next generation is a linear combination of the distribution your current generation generates and external data. As the amount of external data goes to zero, you expect model collapse. This is hardly surprising. I don't know anyone who expects you can just keep training based on previous results and expect something radically new to happen. (Though something *useful* can happen - eg. you may improve performance this way. See "rectification" in flow-matching.)
Note that this doesn't rule out all forms of self-training - just one kind. As a concrete example, an LLM trained to generate code can learn from the output of the generated code. Such output is, in some sense, exogenous.
@dpiponi @Quantensalat Yep, i also did imply this on my post's last remarks. https://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/#:~:text=The%20smarter%20path%20–%20and%20what%20labs%20are%20quietly%20shifting%20toward%20–%20is%C2%A0better%20data%2C%20better%20curation%2C%20better%20grounding%20in%20reality.%20Which%2C%20ironically%2C%20means%20humans%20stay%20in%20the%20loop%20longer%20than%20the%20singularitarians%20wanted.
-
@devsimsek this only means that LLMs can't provide their own training data, right? Could they still "invent" new algorithms, that make more of the existing data?
@laalsaas yep
-
@devsimsek Nobody ever claimed that llms get better by being trained on their own synthetic data. This blog post is very misleading.
The idea of self-improvement and singularity is that llms write improved versions of their own codebase and perform the research and experiments for coming up with better models themselves.
The idea of singularity is interesting but also full of hidden assumptions. I'm always confused when people act like singularity would exist. It's just science fiction.@kaidu Sure, the title is satirical, but I don't think that you have done a great job while reading it. Since both the article and my post specifically talk about one of the training methods...
-
@dpiponi @Quantensalat Yep, i also did imply this on my post's last remarks. https://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/#:~:text=The%20smarter%20path%20–%20and%20what%20labs%20are%20quietly%20shifting%20toward%20–%20is%C2%A0better%20data%2C%20better%20curation%2C%20better%20grounding%20in%20reality.%20Which%2C%20ironically%2C%20means%20humans%20stay%20in%20the%20loop%20longer%20than%20the%20singularitarians%20wanted.
@devsimsek @Quantensalat Yeah, I did kinda guess that's what you meant by "better grounding in reality" although it could also mean real reality
-
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about ithttps://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
@devsimsek The Habsburgs had a better chance of evolving into superhumans.
-
@devsimsek if i eat my own shit repeatedly will i become a singularity
@kieraaa I don't know, someone should simulate that.
-
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about ithttps://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
They so want AI to evolve like humans did, but faster. But on the individual timescale intelligence is a temporary affliction. The body and mind deteriorate. And there's no Moore's Law for neurons so good luck brute forcing billionaire intelligence.
-
They so want AI to evolve like humans did, but faster. But on the individual timescale intelligence is a temporary affliction. The body and mind deteriorate. And there's no Moore's Law for neurons so good luck brute forcing billionaire intelligence.
@aka_quant_noir @devsimsek Oh I think we've achieved billionaire intelligence already. I just have a much dimmer view of billionaires.
-
> Human-generated data is irreplaceable. The “internet is running out of training data” problem just got mathematically formalized.
Yeah I think the AI con mob has realized this already (but of course not saying the quiet part out loud). With Satya whining about people calling it slop and the AI industry trying to force it down everyone's throats no matter the cost (e.g. Copilot) I think they realize that there is only so much internet and historical content they can use to train their models - now they want *you* to help train it for them. Prompt Claude to spit out some code, ask Copilot for a PR review, and _interact_ with it, pointing out where it was stupid, confirming when it did a good job, by virtue of interacting with an AI model you are improving it with this exact, essential human input.
@flaki And it's why companies like Atlassian keep sending out notices that they're going to start using all of the data you've been forced to put on their servers because they took away local licensing, and feeding it into their ditto machines.
