@devsimsek @qualia I think you claim too much here. As I understand it, this result deals only with the intrinsic failures of RL-flavored approaches and not things like self-play, let alone problems that might arise from merely very good AI that still outdoes humans economically.

And I largely agree! I'm glad that someone's finally formalized the intuition that synthetic data is sawdust to bulk out real-world data with and more carefully investigated catastrophic forgetting and the general weaknesses of gradient descent.

That said... to what extent did you have Claude write this post? Because the format is... distinctive.