Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
-
@devsimsek @dpiponi that they act like AI=LLMs?
@Quantensalat @devsimsek There's a setup around equations (9) and (10) where the distribution used for training the next generation is a linear combination of the distribution your current generation generates and external data. As the amount of external data goes to zero, you expect model collapse. This is hardly surprising. I don't know anyone who expects you can just keep training based on previous results and expect something radically new to happen. (Though something *useful* can happen - eg. you may improve performance this way. See "rectification" in flow-matching.)
Note that this doesn't rule out all forms of self-training - just one kind. As a concrete example, an LLM trained to generate code can learn from the output of the generated code. Such output is, in some sense, exogenous.
-
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about it
https://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
> Human-generated data is irreplaceable. The “internet is running out of training data” problem just got mathematically formalized.
Yeah I think the AI con mob has realized this already (but of course not saying the quiet part out loud). With Satya whining about people calling it slop and the AI industry trying to force it down everyone's throats no matter the cost (e.g. Copilot) I think they realize that there is only so much internet and historical content they can use to train their models - now they want *you* to help train it for them. Prompt Claude to spit out some code, ask Copilot for a PR review, and _interact_ with it, pointing out where it was stupid, confirming when it did a good job, by virtue of interacting with an AI model you are improving it with this exact, essential human input.
-
@Quantensalat @devsimsek There's a setup around equations (9) and (10) where the distribution used for training the next generation is a linear combination of the distribution your current generation generates and external data. As the amount of external data goes to zero, you expect model collapse. This is hardly surprising. I don't know anyone who expects you can just keep training based on previous results and expect something radically new to happen. (Though something *useful* can happen - eg. you may improve performance this way. See "rectification" in flow-matching.)
Note that this doesn't rule out all forms of self-training - just one kind. As a concrete example, an LLM trained to generate code can learn from the output of the generated code. Such output is, in some sense, exogenous.
@dpiponi @Quantensalat Yep, i also did imply this on my post's last remarks. https://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/#:~:text=The%20smarter%20path%20–%20and%20what%20labs%20are%20quietly%20shifting%20toward%20–%20is%C2%A0better%20data%2C%20better%20curation%2C%20better%20grounding%20in%20reality.%20Which%2C%20ironically%2C%20means%20humans%20stay%20in%20the%20loop%20longer%20than%20the%20singularitarians%20wanted.
-
@devsimsek this only means that LLMs can't provide their own training data, right? Could they still "invent" new algorithms, that make more of the existing data?
@laalsaas yep
-
@devsimsek Nobody ever claimed that llms get better by being trained on their own synthetic data. This blog post is very misleading.
The idea of self-improvement and singularity is that llms write improved versions of their own codebase and perform the research and experiments for coming up with better models themselves.
The idea of singularity is interesting but also full of hidden assumptions. I'm always confused when people act like singularity would exist. It's just science fiction.@kaidu Sure, the title is satirical, but I don't think that you have done a great job while reading it. Since both the article and my post specifically talk about one of the training methods...
-
@dpiponi @Quantensalat Yep, i also did imply this on my post's last remarks. https://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/#:~:text=The%20smarter%20path%20–%20and%20what%20labs%20are%20quietly%20shifting%20toward%20–%20is%C2%A0better%20data%2C%20better%20curation%2C%20better%20grounding%20in%20reality.%20Which%2C%20ironically%2C%20means%20humans%20stay%20in%20the%20loop%20longer%20than%20the%20singularitarians%20wanted.
@devsimsek @Quantensalat Yeah, I did kinda guess that's what you meant by "better grounding in reality" although it could also mean real reality
-
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about ithttps://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
@devsimsek The Habsburgs had a better chance of evolving into superhumans.
-
@devsimsek if i eat my own shit repeatedly will i become a singularity
@kieraaa I don't know, someone should simulate that.
-
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about ithttps://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
They so want AI to evolve like humans did, but faster. But on the individual timescale intelligence is a temporary affliction. The body and mind deteriorate. And there's no Moore's Law for neurons so good luck brute forcing billionaire intelligence.
-
They so want AI to evolve like humans did, but faster. But on the individual timescale intelligence is a temporary affliction. The body and mind deteriorate. And there's no Moore's Law for neurons so good luck brute forcing billionaire intelligence.
@aka_quant_noir @devsimsek Oh I think we've achieved billionaire intelligence already. I just have a much dimmer view of billionaires.
-
> Human-generated data is irreplaceable. The “internet is running out of training data” problem just got mathematically formalized.
Yeah I think the AI con mob has realized this already (but of course not saying the quiet part out loud). With Satya whining about people calling it slop and the AI industry trying to force it down everyone's throats no matter the cost (e.g. Copilot) I think they realize that there is only so much internet and historical content they can use to train their models - now they want *you* to help train it for them. Prompt Claude to spit out some code, ask Copilot for a PR review, and _interact_ with it, pointing out where it was stupid, confirming when it did a good job, by virtue of interacting with an AI model you are improving it with this exact, essential human input.
@flaki And it's why companies like Atlassian keep sending out notices that they're going to start using all of the data you've been forced to put on their servers because they took away local licensing, and feeding it into their ditto machines.
-
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about ithttps://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
@devsimsek and this is old math, old theory, old knowledge. Gods do I wish I'd kept the various papers.
We've literally known for over two decades that LLMs are dead-ends. It's why IBM spent billions hyper-focusing Watson. We already knew more context just made it worse, regardless of compute or method. It's not 'intelligence,' it's a bad search function. There's shit demonstrating that back to the 1980's.
-
@devsimsek I think AGI and self-improvement is possible. But definitely not with the technology (neural LLMs) that is being marketed as "AI" today.
I think that AGI needs to be able to think logically.@devsimsek@universeodon.com @LunaDragofelis@void.lgbt
if you make agi able to think logically then the world ends.
we need to stop all ai research. if you are researching ai, and are not actively trying to sabotage it, then everyone's going to die. -
@Quantensalat @devsimsek There's a setup around equations (9) and (10) where the distribution used for training the next generation is a linear combination of the distribution your current generation generates and external data. As the amount of external data goes to zero, you expect model collapse. This is hardly surprising. I don't know anyone who expects you can just keep training based on previous results and expect something radically new to happen. (Though something *useful* can happen - eg. you may improve performance this way. See "rectification" in flow-matching.)
Note that this doesn't rule out all forms of self-training - just one kind. As a concrete example, an LLM trained to generate code can learn from the output of the generated code. Such output is, in some sense, exogenous.
@dpiponi @Quantensalat @devsimsek that part, that is ultimately a rehash of well-known theory. THAT part IIRC goes back to like the 1940's or 1950's.
And it absolutely rules out all forms of 'self-training.' It is not just mathematically impossible but a total logical fallacy. How can a system with no reference make correct determinations? Simple: it can't.
-
@anne_twain @devsimsek this requires two components LLMs do not, cannot, and will not ever have. Intent and originality.
Researchers have done self-modifying targeted things. It takes no time at all for things to become impossible for humans to understand. This does not mean they are better. Usually they weren't. Even when hyper-focused with specific controls. -
@devsimsek this is one of those things that seemed intuitive to us skeptics but it's great to see it proven
@huxley @devsimsek doesn't scepticism and intuation mitigate each other?
-
@aka_quant_noir @devsimsek Oh I think we've achieved billionaire intelligence already. I just have a much dimmer view of billionaires.
@alahmnat @devsimsek
I think we're in the billionaire intelligence decline phase. They're going nuts. -
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about ithttps://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
@devsimsek excellent. Thanks for the overview!
-
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about ithttps://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
@devsimsek isn't the idea of self-improving AI that the AI modifies its code, so the underlying algorithm / architecture?
-
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about ithttps://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
@devsimsek @qualia I think you claim too much here. As I understand it, this result deals only with the intrinsic failures of RL-flavored approaches and not things like self-play, let alone problems that might arise from merely very good AI that still outdoes humans economically.
And I largely agree! I'm glad that someone's finally formalized the intuition that synthetic data is sawdust to bulk out real-world data with and more carefully investigated catastrophic forgetting and the general weaknesses of gradient descent.
That said... to what extent did you have Claude write this post? Because the format is... distinctive.
