I said I *didn't* think Taylor Lorenz was an industry shill but... wow... now I'm starting to rethink that.
-
We were thinking about what would happen if the whole AI industry vanished and someone mentioned "cyber security" ... but all of the problems they solve are the ones they created.
AI industry: "this is our greatest success"
The greatest success: Finding [potential bugs] i.e. bits of code, to an extent that is a problem in itself, drowning maintainers in reports. -
@futurebird the watermarking is also the result of some EU regulations I think.
Also some LLM enjoyers are a little bit delusional. They will generate a large document using LLMs wholesale and genuinely believe that whatever comes out is the product of their brilliant mind because they composed the prompt.
@loathsome_dongeater @futurebird There are a lot of people who secretly hate the activity of making things, who have to make things as a chore related to pursuing a career; it's a real gift to them, as is all the aggressive rhetoric in support of the systems normalised by the vendors and their huge bags of PR money.
-
But, why not just be honest and say you used and LLM? Why so bashful?
In my experience, it's because they know most people absolutely hate having slop sprayed at their face, so they'll go to great lengths to disguise it in an attempt to "get one over" on them.
It resembles the same type of shit with people who "test" someone else's allergies by sneaking stuff into their food: They're trying to "catch them in a lie" with the false assumption that they'll be fine with it if they don't know it's there.
And even just that type of behavior is extremely concerning for me, even if it doesn't involve allergens that can Actually Kill People. It shows a clear lack of respect for others, and deserves a swift football kick to the unmentionables and immediate expulsion.
Isn't that amazing? Even people who really love boosting AI feel hurt when someone feeds them AI generated content.
It's embarrassing to admit to liking or not noticing AI generated content. There was a music video I really enjoyed a few months back and I think it might have AI generated music or AI assisted animation. The creator hasn't been very transparent and everyone who liked it is kind of worried and unhappy.
When the same account posted something new I ignored it.
-
Isn't that amazing? Even people who really love boosting AI feel hurt when someone feeds them AI generated content.
It's embarrassing to admit to liking or not noticing AI generated content. There was a music video I really enjoyed a few months back and I think it might have AI generated music or AI assisted animation. The creator hasn't been very transparent and everyone who liked it is kind of worried and unhappy.
When the same account posted something new I ignored it.
This is the song/creator
I even thought about taking this post down because I don't really know. But, I decided not to and to simply wait to see what was really going on.
I still don't know. But, due to generated content there is a cloud hanging over many creative ventures, fairly or not.
-
@futurebird My understanding is that it's mostly happening because the EU requires it.
Pre-training datasets are pretty carefully curated and they have strong quality filters, so I don't think there is much of a risk of model collapse in practice.
It's also not so much about special characters or word choice as about the choices between equally good options (by the models' own probabilities). Every time you choose between two equally likely options you are essentially "encoding" one bit (i.e. a coinflip) of information. You can use these bits to create a watermark without losing quality.
@pbloem @futurebird Yeah, none of that. If datasets were curated we wouldn't get glue on pizza & vendors are already fretting about possible model collapse. Also, "watermarking", which is really fingerprinting, can't possibly work as advertised & even if it did it's trivial to defeat & per OpenAI is only correct a measly 99.9% of the time, which would mean thousands upon thousands of errors every day...
-
The companies that run LLMs can also use this for PR to calm concerns from the public about the proliferation of such content.
From a CS perspective I can't think of any way to have a watermark that couldn't be easily defeated through additional processing. LLM dependent people currently take pains to remove em dashes so this new wrinkle has some of them in a panic.
But, why not just be honest and say you used and LLM? Why so bashful?
From a CS perspective I can't think of any way to have a watermark that couldn't be easily defeated through additional processing
The way that the current ones work is that they tweak the probabilities to generate specific patterns in the output. Where there's an equal probability of two words following another in the raw weights, they'll tune it so that there's a higher probability of one than the other.
If you know the weights and know the biassing factor, you can look at each word pair and see what the probability would be of the model generating that.
This means that the watermark is smeared all over the output. And it's not a binary thing though, each pair of word contributes something to the probability of matching the watermark and looking at the whole thing will give you a probability at the end.
Changing every other word should give you close to a 0% of matching the watermark but, at that point, why bother using the LLM at all? If you're going to change half of the words, you may as well just write them yourself. And that's a problem for people who want to share low-effort slop and pretend to be creative.
-
She platformed Chaya Raichik with an interview that really didn't challenge the bigot. I also saw a personal story about Lorenz using dodgy practices at the NYT to build her portfolio by stealing from the sources and beat of others.
-
From a CS perspective I can't think of any way to have a watermark that couldn't be easily defeated through additional processing
The way that the current ones work is that they tweak the probabilities to generate specific patterns in the output. Where there's an equal probability of two words following another in the raw weights, they'll tune it so that there's a higher probability of one than the other.
If you know the weights and know the biassing factor, you can look at each word pair and see what the probability would be of the model generating that.
This means that the watermark is smeared all over the output. And it's not a binary thing though, each pair of word contributes something to the probability of matching the watermark and looking at the whole thing will give you a probability at the end.
Changing every other word should give you close to a 0% of matching the watermark but, at that point, why bother using the LLM at all? If you're going to change half of the words, you may as well just write them yourself. And that's a problem for people who want to share low-effort slop and pretend to be creative.
@david_chisnall @futurebird @404mediaco has a pretty good article OpEd on it as well.
-
This is the song/creator
I even thought about taking this post down because I don't really know. But, I decided not to and to simply wait to see what was really going on.
I still don't know. But, due to generated content there is a cloud hanging over many creative ventures, fairly or not.
-
@pbloem @futurebird Yeah, none of that. If datasets were curated we wouldn't get glue on pizza & vendors are already fretting about possible model collapse. Also, "watermarking", which is really fingerprinting, can't possibly work as advertised & even if it did it's trivial to defeat & per OpenAI is only correct a measly 99.9% of the time, which would mean thousands upon thousands of errors every day...
Let me go point by point here.
- Curation doesn't mean manual selection. It just means that you don't blindly ingest everything from the internet. This has been done since GPT2, when they used social media upvotes as a proxy for article quality. In GPT3, they included sources proportional to the quality of the source which gives you the best of both worlds.
- All sorts of junk sneaks into the data, but so long as the majority is high quality, it won't lead to model collapse.
- "Glue on Pizza" came from the Google AI summary which is a poor proxy for current AI capability.
- Watermarking can and does 100% work as advertised. For a simple insight to the idea, consider the bitstream used to feed the PRNG. If you have the complete model output of one session and the model, you can reconstruct that. Save it, use the same stream each session and you have your watermark with no changes to the model (some extra tricks are necessary to watermark subsets of the conversation).
- It is indeed relatively easy to circumvent, although the precise method they use seems to be somewhat robust to basic rephrasing. Still, it's going to catch out a lot of lazy people. -
But, why not just be honest and say you used and LLM? Why so bashful?
In my experience, it's because they know most people absolutely hate having slop sprayed at their face, so they'll go to great lengths to disguise it in an attempt to "get one over" on them.
It resembles the same type of shit with people who "test" someone else's allergies by sneaking stuff into their food: They're trying to "catch them in a lie" with the false assumption that they'll be fine with it if they don't know it's there.
And even just that type of behavior is extremely concerning for me, even if it doesn't involve allergens that can Actually Kill People. It shows a clear lack of respect for others, and deserves a swift football kick to the unmentionables and immediate expulsion.
@becomethewaifu @futurebird the annoying thing is that people who "sneak ai into your food" can get away with it more easily, if i dont notice then i wont have an allergic reaction (ngl it would be cool if i could detect it that way) so they think they have proven you wrong when that is not the point
-
This is the song/creator
I even thought about taking this post down because I don't really know. But, I decided not to and to simply wait to see what was really going on.
I still don't know. But, due to generated content there is a cloud hanging over many creative ventures, fairly or not.
@futurebird @becomethewaifu It’s funny, there’s so much accusation of AI around that video and to me it has none of the hallmarks and in fact looks far more likely to be human created (there’s several animated sequences that are exact repeats, which is unlike AI) but I think “simply styled animation” reads as AI.
There’s a comic on here that keeps getting accusations of being AI generated when it’s likely the artist’s work has been used as primary training material, and he’s really unhappy about it.
-
@futurebird @becomethewaifu It’s funny, there’s so much accusation of AI around that video and to me it has none of the hallmarks and in fact looks far more likely to be human created (there’s several animated sequences that are exact repeats, which is unlike AI) but I think “simply styled animation” reads as AI.
There’s a comic on here that keeps getting accusations of being AI generated when it’s likely the artist’s work has been used as primary training material, and he’s really unhappy about it.
@MichaelTBacon @becomethewaifu
I don't think the animation is generated, it's the music that I have questions about. But I've done animation, and I've never really done any music. So I feel more out of my depth and there is something about the vocals and the very good but predictable use of breaks that IDK...
No that's the main point. I Don't Know.
-
The companies that run LLMs can also use this for PR to calm concerns from the public about the proliferation of such content.
From a CS perspective I can't think of any way to have a watermark that couldn't be easily defeated through additional processing. LLM dependent people currently take pains to remove em dashes so this new wrinkle has some of them in a panic.
But, why not just be honest and say you used and LLM? Why so bashful?
@futurebird
On the one hand, this seems like a good idea. It should cut down on academic (and non-academic) cheating among other benefits.But if this really was a benefit to us regular folks, the AI companies would be screaming bloody murder. So they must be getting something out of it. Unless the value of not ingesting their own slop for re-training (and the ensuing model collapse) is worth it all on its own...
-
I said I *didn't* think Taylor Lorenz was an industry shill but... wow... now I'm starting to rethink that.
You know I might be wrong, but what is wrong with watermarks?
@futurebird She's not so much a willing shill (like Newton), but she's addicted to attention, esp once she went independent and it literally pays her bills now. I wrote her off when she expanded from internet culture into deeper waters she doesn't have the chops for.
-
** I think Peter is more correct. The primary driver making this happen is the EU. As an American I forget that this chaos COULD be regulated a little.
I'd like a clear humanly visible one as well.
Like requiring every period (full stop) to be replaced by a middle dot ·.
Still easy to read.
-
@futurebird What’s really wrong with AI watermarks for text is that AI companies believe words are interchangeable for a given meaning. And that only them can theoretically verify the watermark, which feels like a conflict of interest?
@hypolite @futurebird It seems to me that reviewing a text before passing it to others would reveal whether or not it conveys the precise meaning that you intended.
-
I'd like a clear humanly visible one as well.
Like requiring every period (full stop) to be replaced by a middle dot ·.
Still easy to read.
@EricLawton @futurebird Even easier to remove, though.
-
Let me go point by point here.
- Curation doesn't mean manual selection. It just means that you don't blindly ingest everything from the internet. This has been done since GPT2, when they used social media upvotes as a proxy for article quality. In GPT3, they included sources proportional to the quality of the source which gives you the best of both worlds.
- All sorts of junk sneaks into the data, but so long as the majority is high quality, it won't lead to model collapse.
- "Glue on Pizza" came from the Google AI summary which is a poor proxy for current AI capability.
- Watermarking can and does 100% work as advertised. For a simple insight to the idea, consider the bitstream used to feed the PRNG. If you have the complete model output of one session and the model, you can reconstruct that. Save it, use the same stream each session and you have your watermark with no changes to the model (some extra tricks are necessary to watermark subsets of the conversation).
- It is indeed relatively easy to circumvent, although the precise method they use seems to be somewhat robust to basic rephrasing. Still, it's going to catch out a lot of lazy people.@pbloem @futurebird all I hear is "Praise Sam & Dario" on repeat.
-
But, why not just be honest and say you used and LLM? Why so bashful?
In my experience, it's because they know most people absolutely hate having slop sprayed at their face, so they'll go to great lengths to disguise it in an attempt to "get one over" on them.
It resembles the same type of shit with people who "test" someone else's allergies by sneaking stuff into their food: They're trying to "catch them in a lie" with the false assumption that they'll be fine with it if they don't know it's there.
And even just that type of behavior is extremely concerning for me, even if it doesn't involve allergens that can Actually Kill People. It shows a clear lack of respect for others, and deserves a swift football kick to the unmentionables and immediate expulsion.
@becomethewaifu @futurebird I saw a "is this AI?" comment on an old "How it's Made" video. Everything is suspect these days.
