I said I *didn't* think Taylor Lorenz was an industry shill but... wow... now I'm starting to rethink that.
-
** I think Peter is more correct. The primary driver making this happen is the EU. As an American I forget that this chaos COULD be regulated a little.
I'd like a clear humanly visible one as well.
Like requiring every period (full stop) to be replaced by a middle dot ·.
Still easy to read.
-
@futurebird What’s really wrong with AI watermarks for text is that AI companies believe words are interchangeable for a given meaning. And that only them can theoretically verify the watermark, which feels like a conflict of interest?
@hypolite @futurebird It seems to me that reviewing a text before passing it to others would reveal whether or not it conveys the precise meaning that you intended.
-
I'd like a clear humanly visible one as well.
Like requiring every period (full stop) to be replaced by a middle dot ·.
Still easy to read.
@EricLawton @futurebird Even easier to remove, though.
-
Let me go point by point here.
- Curation doesn't mean manual selection. It just means that you don't blindly ingest everything from the internet. This has been done since GPT2, when they used social media upvotes as a proxy for article quality. In GPT3, they included sources proportional to the quality of the source which gives you the best of both worlds.
- All sorts of junk sneaks into the data, but so long as the majority is high quality, it won't lead to model collapse.
- "Glue on Pizza" came from the Google AI summary which is a poor proxy for current AI capability.
- Watermarking can and does 100% work as advertised. For a simple insight to the idea, consider the bitstream used to feed the PRNG. If you have the complete model output of one session and the model, you can reconstruct that. Save it, use the same stream each session and you have your watermark with no changes to the model (some extra tricks are necessary to watermark subsets of the conversation).
- It is indeed relatively easy to circumvent, although the precise method they use seems to be somewhat robust to basic rephrasing. Still, it's going to catch out a lot of lazy people.@pbloem @futurebird all I hear is "Praise Sam & Dario" on repeat.
-
But, why not just be honest and say you used and LLM? Why so bashful?
In my experience, it's because they know most people absolutely hate having slop sprayed at their face, so they'll go to great lengths to disguise it in an attempt to "get one over" on them.
It resembles the same type of shit with people who "test" someone else's allergies by sneaking stuff into their food: They're trying to "catch them in a lie" with the false assumption that they'll be fine with it if they don't know it's there.
And even just that type of behavior is extremely concerning for me, even if it doesn't involve allergens that can Actually Kill People. It shows a clear lack of respect for others, and deserves a swift football kick to the unmentionables and immediate expulsion.
@becomethewaifu @futurebird I saw a "is this AI?" comment on an old "How it's Made" video. Everything is suspect these days.
-
@futurebird there's been more discourse around her before but i forget the context at the moment
@mensrea @futurebird specific to AI, all of the following are recent Lorenz controversies:
- She said she uses ChatGPT for recipes
- She said she spends roughly $300 per month on AI
- She said she enjoys reading some AI generated articles -
From a CS perspective I can't think of any way to have a watermark that couldn't be easily defeated through additional processing
The way that the current ones work is that they tweak the probabilities to generate specific patterns in the output. Where there's an equal probability of two words following another in the raw weights, they'll tune it so that there's a higher probability of one than the other.
If you know the weights and know the biassing factor, you can look at each word pair and see what the probability would be of the model generating that.
This means that the watermark is smeared all over the output. And it's not a binary thing though, each pair of word contributes something to the probability of matching the watermark and looking at the whole thing will give you a probability at the end.
Changing every other word should give you close to a 0% of matching the watermark but, at that point, why bother using the LLM at all? If you're going to change half of the words, you may as well just write them yourself. And that's a problem for people who want to share low-effort slop and pretend to be creative.
@david_chisnall @futurebird this also means that you have know the weights and know the biassing factor to be able read that watermark reliably.
And I am going to bet that Anthropic is not going to release those.
In other words, Anthropic will be the only company offering the slop generator, and the service to tell that something got generated by it.
That's real power right there. Would they not "tweak" the results if it was useful for them, say, to create a plagiarism scandal around someone?
-
Isn't that amazing? Even people who really love boosting AI feel hurt when someone feeds them AI generated content.
It's embarrassing to admit to liking or not noticing AI generated content. There was a music video I really enjoyed a few months back and I think it might have AI generated music or AI assisted animation. The creator hasn't been very transparent and everyone who liked it is kind of worried and unhappy.
When the same account posted something new I ignored it.
@futurebird
Frugit
Can't share until I know.
-
@mensrea @futurebird specific to AI, all of the following are recent Lorenz controversies:
- She said she uses ChatGPT for recipes
- She said she spends roughly $300 per month on AI
- She said she enjoys reading some AI generated articles@gryphonmyers@mastodon.social @futurebird@sauropods.win @mensrea@freeradical.zone had to look up the $300 a month figure and yup (sorry it's a substack link) https://on.substack.com/p/model-behavior-taylor-lorenz
-
@david_chisnall @futurebird this also means that you have know the weights and know the biassing factor to be able read that watermark reliably.
And I am going to bet that Anthropic is not going to release those.
In other words, Anthropic will be the only company offering the slop generator, and the service to tell that something got generated by it.
That's real power right there. Would they not "tweak" the results if it was useful for them, say, to create a plagiarism scandal around someone?
Yup, that's a legitimate concern. I wouldn't be surprised if they've been doing this for a while anyway to avoid training on their own output, but releasing an API to query gives them a lot of control.
And, of course, you can't detect slop, you can detect slop generated by a specific tool.
-
@futurebird What’s really wrong with AI watermarks for text is that AI companies believe words are interchangeable for a given meaning. And that only them can theoretically verify the watermark, which feels like a conflict of interest?@hypolite @futurebird remember, it's nothing to do with meaning - it's everything to do with corpus correlation.
-
@MichaelTBacon @becomethewaifu
I don't think the animation is generated, it's the music that I have questions about. But I've done animation, and I've never really done any music. So I feel more out of my depth and there is something about the vocals and the very good but predictable use of breaks that IDK...
No that's the main point. I Don't Know.
Up front, I really don't know either.
I have done a good bit more music than animation, and there's certainly nothing in that that couldn't have been done with a garden variety synth/electric piano setup. The only thing really tricky is that the drum and the bass are really tight, which really propels the song. (The lyrics are clever and hilarious and well timed as well but it's the rhythm section that makes it so re-listenable.)
I guess the thing that's so annoying in this instance is that all of it *could* have been done by a dude who was pretty good with basic animation skills, a synth setup, and a friend who had been a bass player in a local scene for a couple decades. But maybe it could have been done by AI?
At this point, I don't want AI watermarks so much as I want composable digital signatures for real work. Like, a signature from the mixer that this feed came in off an analog feed.
Just as you say. I Don't Know.
-
I said I *didn't* think Taylor Lorenz was an industry shill but... wow... now I'm starting to rethink that.
You know I might be wrong, but what is wrong with watermarks?
@futurebird that doesn't surprise me, I think the only thing she really cares about is climbing the ladder of self promotion. I blocked her years ago because o got a bad vibe from her, she has more in common with Bari Weiss or Glenn Greenwald than a real journalist
-
I said I *didn't* think Taylor Lorenz was an industry shill but... wow... now I'm starting to rethink that.
You know I might be wrong, but what is wrong with watermarks?
@futurebird I don't like the idea of watermarking because it's going to allow companies to make money in the slop creation and the slop detection sides of things - anyone who wants to detect AI will need to pay EVERY AI Company to check against their own watermarking keys, which means only the rich will be able to detect slop. Plus from what I have read detecting the slop is as computationally expensive as producing the slop in the first place, so it could double the impacts we are seeing.
-
J jwcph@helvede.net shared this topic
-
@futurebird I don't like the idea of watermarking because it's going to allow companies to make money in the slop creation and the slop detection sides of things - anyone who wants to detect AI will need to pay EVERY AI Company to check against their own watermarking keys, which means only the rich will be able to detect slop. Plus from what I have read detecting the slop is as computationally expensive as producing the slop in the first place, so it could double the impacts we are seeing.
@futurebird none of this has anything to do with being tracked, or being exposed as a fraud or whatever. I just hate that companies get to make money to make the mess AND to be the "savior" who cleans it up
-
I said I *didn't* think Taylor Lorenz was an industry shill but... wow... now I'm starting to rethink that.
You know I might be wrong, but what is wrong with watermarks?
@futurebird@sauropods.win I gotta watch the video first but what did she say that makes you think that?
-
I said I *didn't* think Taylor Lorenz was an industry shill but... wow... now I'm starting to rethink that.
You know I might be wrong, but what is wrong with watermarks?
@futurebird I've never gotten "shill" off her, but her leaning into youtuber stereotypes really put me off her reporting, which I used to quite enjoy.
-
@futurebird@sauropods.win I gotta watch the video first but what did she say that makes you think that?
The negative take on AI watermarking surprised me. That's all. Watermarks seem responsible?
-
@pbloem @futurebird all I hear is "Praise Sam & Dario" on repeat.
@jwcph @futurebird I know. It's not what I'm saying or thinking though.
-
Let me outline my understanding of "LLM Watermarking"
It is possible to embed special characters and patterns in the output of LLMs that would make text generated by these systems easier to reliably detect. This is mainly being done so that when LLMs scrape the web for new information they can avoid ingesting machine generated content.**
When you train an LLM on machine generated content it may lead to "model collapse."
**see next post for correction
@futurebird My (weak) understanding of AI Text watermarking is that they tweak the probabilities of words in certain positions so they're more likely to line up with complex watermark patterns.
I wonder if that will make their already noticeable style more or less recognizable.
I'll take their word for it that the watermark patterns aren't humanly detectable, but the fact that they're deliberately choosing slightly non-optimal words and phrases must have an effect?
