I said I *didn't* think Taylor Lorenz was an industry shill but... wow... now I'm starting to rethink that.
-
I said I *didn't* think Taylor Lorenz was an industry shill but... wow... now I'm starting to rethink that.
You know I might be wrong, but what is wrong with watermarks?
@futurebird there's been more discourse around her before but i forget the context at the moment
-
I said I *didn't* think Taylor Lorenz was an industry shill but... wow... now I'm starting to rethink that.
You know I might be wrong, but what is wrong with watermarks?
Let me outline my understanding of "LLM Watermarking"
It is possible to embed special characters and patterns in the output of LLMs that would make text generated by these systems easier to reliably detect. This is mainly being done so that when LLMs scrape the web for new information they can avoid ingesting machine generated content.**
When you train an LLM on machine generated content it may lead to "model collapse."
**see next post for correction
-
I said I *didn't* think Taylor Lorenz was an industry shill but... wow... now I'm starting to rethink that.
You know I might be wrong, but what is wrong with watermarks?
@futurebird so I find myself generally in the pro column, but I can see two reasons against:
first, it does provide a certain level of tracking, not unlike that would just put in into our printers to identify exactly which printer something came from.
I can also see it as a real boon for the companies so that they don’t choke on re-uploading their output and get worse, which is a known problem
-
I said I *didn't* think Taylor Lorenz was an industry shill but... wow... now I'm starting to rethink that.
You know I might be wrong, but what is wrong with watermarks?
@futurebird Not sure on any of that but Watermarking seems to be a different issue.
https://www.404media.co/anthropics-text-watermarking-proves-ai-companies-do-not-care-at-all-about-writing/ -
I said I *didn't* think Taylor Lorenz was an industry shill but... wow... now I'm starting to rethink that.
You know I might be wrong, but what is wrong with watermarks?
@futurebird What’s really wrong with AI watermarks for text is that AI companies believe words are interchangeable for a given meaning. And that only them can theoretically verify the watermark, which feels like a conflict of interest? -
Let me outline my understanding of "LLM Watermarking"
It is possible to embed special characters and patterns in the output of LLMs that would make text generated by these systems easier to reliably detect. This is mainly being done so that when LLMs scrape the web for new information they can avoid ingesting machine generated content.**
When you train an LLM on machine generated content it may lead to "model collapse."
**see next post for correction
The companies that run LLMs can also use this for PR to calm concerns from the public about the proliferation of such content.
From a CS perspective I can't think of any way to have a watermark that couldn't be easily defeated through additional processing. LLM dependent people currently take pains to remove em dashes so this new wrinkle has some of them in a panic.
But, why not just be honest and say you used and LLM? Why so bashful?
-
@futurebird so I find myself generally in the pro column, but I can see two reasons against:
first, it does provide a certain level of tracking, not unlike that would just put in into our printers to identify exactly which printer something came from.
I can also see it as a real boon for the companies so that they don’t choke on re-uploading their output and get worse, which is a known problem
@futurebird in the first case I am extremely anti-printer finger marking so I can be sympathetic to it… But since AI is used for widescale fraud or deceptive practices ot is not based on the government wanting to identify a person but based on supporting individuals trying to make sense of this extremely slopified world
but the second, I think it’s the reason that the AI companies are willing to do it without a bigger fuss. They realized they can use it to eliminate bad training data.
-
The companies that run LLMs can also use this for PR to calm concerns from the public about the proliferation of such content.
From a CS perspective I can't think of any way to have a watermark that couldn't be easily defeated through additional processing. LLM dependent people currently take pains to remove em dashes so this new wrinkle has some of them in a panic.
But, why not just be honest and say you used and LLM? Why so bashful?
@futurebird What I expect from AI companies is Malicious Compliance.
And Malicious Non-Compliance. -
I said I *didn't* think Taylor Lorenz was an industry shill but... wow... now I'm starting to rethink that.
You know I might be wrong, but what is wrong with watermarks?
@futurebird @kim_harding Even before it becomes easy to detect the watermark itself, the noise that AI users make about it is an excellent detector for AI content.
-
The companies that run LLMs can also use this for PR to calm concerns from the public about the proliferation of such content.
From a CS perspective I can't think of any way to have a watermark that couldn't be easily defeated through additional processing. LLM dependent people currently take pains to remove em dashes so this new wrinkle has some of them in a panic.
But, why not just be honest and say you used and LLM? Why so bashful?
@futurebird I remember when Genius.com accused Google of plagiarizing their song lyrics and as evidence showed that the weird spacing and capitalization they had intentionally added to a song were reproduced 1:1 on Google's uncredited copy
so I think the idea is that people will be too lazy to try to defeat the "watermarking"? but that's clearly already not the case for LLM slop
-
The companies that run LLMs can also use this for PR to calm concerns from the public about the proliferation of such content.
From a CS perspective I can't think of any way to have a watermark that couldn't be easily defeated through additional processing. LLM dependent people currently take pains to remove em dashes so this new wrinkle has some of them in a panic.
But, why not just be honest and say you used and LLM? Why so bashful?
@futurebird Honestly it would make detecting AI plagerism/assigment writing easier.
-
Let me outline my understanding of "LLM Watermarking"
It is possible to embed special characters and patterns in the output of LLMs that would make text generated by these systems easier to reliably detect. This is mainly being done so that when LLMs scrape the web for new information they can avoid ingesting machine generated content.**
When you train an LLM on machine generated content it may lead to "model collapse."
**see next post for correction
@futurebird My understanding is that it's mostly happening because the EU requires it.
Pre-training datasets are pretty carefully curated and they have strong quality filters, so I don't think there is much of a risk of model collapse in practice.
It's also not so much about special characters or word choice as about the choices between equally good options (by the models' own probabilities). Every time you choose between two equally likely options you are essentially "encoding" one bit (i.e. a coinflip) of information. You can use these bits to create a watermark without losing quality.
-
@futurebird My understanding is that it's mostly happening because the EU requires it.
Pre-training datasets are pretty carefully curated and they have strong quality filters, so I don't think there is much of a risk of model collapse in practice.
It's also not so much about special characters or word choice as about the choices between equally good options (by the models' own probabilities). Every time you choose between two equally likely options you are essentially "encoding" one bit (i.e. a coinflip) of information. You can use these bits to create a watermark without losing quality.
@pbloem @futurebird Yes it seems very much driven by the EU AI act, presumably for general transparency rather than specifically about AI training.
-
@futurebird I remember when Genius.com accused Google of plagiarizing their song lyrics and as evidence showed that the weird spacing and capitalization they had intentionally added to a song were reproduced 1:1 on Google's uncredited copy
so I think the idea is that people will be too lazy to try to defeat the "watermarking"? but that's clearly already not the case for LLM slop
People using slop have self-selected to be too lazy to defeat anything.
But "just use our secret decoder to prove that someone used our secret coder" is a self-serving non-solution.
-
@futurebird I remember when Genius.com accused Google of plagiarizing their song lyrics and as evidence showed that the weird spacing and capitalization they had intentionally added to a song were reproduced 1:1 on Google's uncredited copy
so I think the idea is that people will be too lazy to try to defeat the "watermarking"? but that's clearly already not the case for LLM slop
@futurebird like if there's one discussion about watermarks I've seen on the internet ever, it's artists saying "the malicious person cropped out my signature"
the one thing I know about watermarks is that they are easily defeated by determined assholes
-
The companies that run LLMs can also use this for PR to calm concerns from the public about the proliferation of such content.
From a CS perspective I can't think of any way to have a watermark that couldn't be easily defeated through additional processing. LLM dependent people currently take pains to remove em dashes so this new wrinkle has some of them in a panic.
But, why not just be honest and say you used and LLM? Why so bashful?
@futurebird the way certain people are reacting to the watermarks—which have been in place for a little while already—is very telling.
-
@futurebird there's been more discourse around her before but i forget the context at the moment
-
Let me outline my understanding of "LLM Watermarking"
It is possible to embed special characters and patterns in the output of LLMs that would make text generated by these systems easier to reliably detect. This is mainly being done so that when LLMs scrape the web for new information they can avoid ingesting machine generated content.**
When you train an LLM on machine generated content it may lead to "model collapse."
**see next post for correction
** I think Peter is more correct. The primary driver making this happen is the EU. As an American I forget that this chaos COULD be regulated a little.
-
The companies that run LLMs can also use this for PR to calm concerns from the public about the proliferation of such content.
From a CS perspective I can't think of any way to have a watermark that couldn't be easily defeated through additional processing. LLM dependent people currently take pains to remove em dashes so this new wrinkle has some of them in a panic.
But, why not just be honest and say you used and LLM? Why so bashful?
@futurebird excellent explainer about watermarking in text: https://declaude.org/watermarking/
-
I said I *didn't* think Taylor Lorenz was an industry shill but... wow... now I'm starting to rethink that.
You know I might be wrong, but what is wrong with watermarks?
@futurebird Oh, she's been one of the bad ones for a long time...
