@jwcph @futurebird I know. It's not what I'm saying or thinking though.

@jwcph @futurebird

Let me go point by point here.
- Curation doesn't mean manual selection. It just means that you don't blindly ingest everything from the internet. This has been done since GPT2, when they used social media upvotes as a proxy for article quality. In GPT3, they included sources proportional to the quality of the source which gives you the best of both worlds.
- All sorts of junk sneaks into the data, but so long as the majority is high quality, it won't lead to model collapse.
- "Glue on Pizza" came from the Google AI summary which is a poor proxy for current AI capability.
- Watermarking can and does 100% work as advertised. For a simple insight to the idea, consider the bitstream used to feed the PRNG. If you have the complete model output of one session and the model, you can reconstruct that. Save it, use the same stream each session and you have your watermark with no changes to the model (some extra tricks are necessary to watermark subsets of the conversation).
- It is indeed relatively easy to circumvent, although the precise method they use seems to be somewhat robust to basic rephrasing. Still, it's going to catch out a lot of lazy people.

@futurebird My understanding is that it's mostly happening because the EU requires it.

Pre-training datasets are pretty carefully curated and they have strong quality filters, so I don't think there is much of a risk of model collapse in practice.

It's also not so much about special characters or word choice as about the choices between equally good options (by the models' own probabilities). Every time you choose between two equally likely options you are essentially "encoding" one bit (i.e. a coinflip) of information. You can use these bits to create a watermark without losing quality.