Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
-
@devsimsek Is that a thing people believe, that LLMs generate themselves towards the singularity simply by eating their own output and no other feedback?
@Quantensalat @devsimsek I'm sure you'll find plenty of straw men who do
-
@devsimsek Chatting with U Toronto AI profs 6, 7 years ago, I posed a problem.
"Teach your AI everything about whole, integer, rational and real numbers. Ask it to solve a problem that requires it to invent complex numbers."
Reply: "Oh... It doesn't work that way."
I knew that, but the ability to frame your observations as the product of a higher order system is IMHO key to what we call "intelligence". Collecting evidence that can disprove your hypothesis is science.
LLM approaches are neither, in a very expensive way.
I'll have to read the paper, though. I'm looking forward to the AI equivalent of Goedel's Theorem that shuts down this annoying iteration of the field.
@TallSimon @devsimsek I haven’t looked at the proof, but I wonder if Gödel plays a role in it. Seems like at least Gödel would strongly imply this new proof.
-
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about it
https://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
@devsimsek this is one of those things that seemed intuitive to us skeptics but it's great to see it proven
-
@Quantensalat @devsimsek I'm sure you'll find plenty of straw men who do
@dpiponi @devsimsek I find the paper interesting but I would like to understand the exact
premises. "AI" is not equal to gen AI or LLMs, it probably makes little sense to sell it as a general statement about "AI" -
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about ithttps://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
@devsimsek wow, almost as if this was a problem known as overtraining for well over 30 years
-
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about ithttps://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
@devsimsek it's the only thing that makes sense if you know just a little about how they work (I don't know more than a little)
Like if you output whatever is most likely, and input that again, it's only logical (at least to me) that eventually you'll get a mushy average -
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about ithttps://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
@devsimsek This feels like a weird argument, because it proves a version that I've never heard anyone arguing for. Like, when I've heard people talk about AI itself accelerating AI's improvement (on both pro and con sides), the argument wasn't that AI would self-train on its own output. The argument was that AI would replace AI developers and accelerate the development of better AI code.
-
@devsimsek wow, almost as if this was a problem known as overtraining for well over 30 years
@SRAZKVT Exactly.
-
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about ithttps://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
@devsimsek Nobody ever claimed that llms get better by being trained on their own synthetic data. This blog post is very misleading.
The idea of self-improvement and singularity is that llms write improved versions of their own codebase and perform the research and experiments for coming up with better models themselves.
The idea of singularity is interesting but also full of hidden assumptions. I'm always confused when people act like singularity would exist. It's just science fiction. -
@dpiponi @devsimsek I find the paper interesting but I would like to understand the exact
premises. "AI" is not equal to gen AI or LLMs, it probably makes little sense to sell it as a general statement about "AI"@Quantensalat @dpiponi That's what I hate about these companies.
-
@devsimsek So... let me get this straight. Autocoprophagic #RSI •doesn't• lead to #AGI? Say it ain't so!
#AI@ghostinthenet Yep
-
@devsimsek also see https://berryvilleiml.com/2026/01/10/recursive-pollution-and-model-collapse-are-not-the-same/
This is part of a long running #ML research thread with big #MLsec impact
@noplasticshower Thanks, ill look into it
-
@anne_twain YEP, most of the people whom commented assumes they develop every iteration with fresh data; that comes from internet ...
-
@Quantensalat @dpiponi That's what I hate about these companies.
@devsimsek @dpiponi that they act like AI=LLMs?
-
@anne_twain @devsimsek
"That's like a high school history class having their own essays as research material." - a memorable phrase. -
@devsimsek @dpiponi that they act like AI=LLMs?
@Quantensalat @dpiponi yes. I did used the same tactic while naming my post as satire. its annoying....
-
@devsimsek @dpiponi that they act like AI=LLMs?
@Quantensalat @devsimsek There's a setup around equations (9) and (10) where the distribution used for training the next generation is a linear combination of the distribution your current generation generates and external data. As the amount of external data goes to zero, you expect model collapse. This is hardly surprising. I don't know anyone who expects you can just keep training based on previous results and expect something radically new to happen. (Though something *useful* can happen - eg. you may improve performance this way. See "rectification" in flow-matching.)
Note that this doesn't rule out all forms of self-training - just one kind. As a concrete example, an LLM trained to generate code can learn from the output of the generated code. Such output is, in some sense, exogenous.
-
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about ithttps://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
> Human-generated data is irreplaceable. The “internet is running out of training data” problem just got mathematically formalized.
Yeah I think the AI con mob has realized this already (but of course not saying the quiet part out loud). With Satya whining about people calling it slop and the AI industry trying to force it down everyone's throats no matter the cost (e.g. Copilot) I think they realize that there is only so much internet and historical content they can use to train their models - now they want *you* to help train it for them. Prompt Claude to spit out some code, ask Copilot for a PR review, and _interact_ with it, pointing out where it was stupid, confirming when it did a good job, by virtue of interacting with an AI model you are improving it with this exact, essential human input.
-
@Quantensalat @devsimsek There's a setup around equations (9) and (10) where the distribution used for training the next generation is a linear combination of the distribution your current generation generates and external data. As the amount of external data goes to zero, you expect model collapse. This is hardly surprising. I don't know anyone who expects you can just keep training based on previous results and expect something radically new to happen. (Though something *useful* can happen - eg. you may improve performance this way. See "rectification" in flow-matching.)
Note that this doesn't rule out all forms of self-training - just one kind. As a concrete example, an LLM trained to generate code can learn from the output of the generated code. Such output is, in some sense, exogenous.
@dpiponi @Quantensalat Yep, i also did imply this on my post's last remarks. https://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/#:~:text=The%20smarter%20path%20–%20and%20what%20labs%20are%20quietly%20shifting%20toward%20–%20is%C2%A0better%20data%2C%20better%20curation%2C%20better%20grounding%20in%20reality.%20Which%2C%20ironically%2C%20means%20humans%20stay%20in%20the%20loop%20longer%20than%20the%20singularitarians%20wanted.
-
@devsimsek this only means that LLMs can't provide their own training data, right? Could they still "invent" new algorithms, that make more of the existing data?
@laalsaas yep
