I've been saying "if AI is making you so productive then where is all this great new software" and I guess the answer is the software is out there it's just not great, it's terrible, and nobody is using it
-
Let's do it like this: Are there limits to how much you can compress data?
(This is a Computer Science 101 question so don't spend too long on it)
I'm not here to answer a measure theory question.
I'm just pointing out an article that goes againsts yours.
-
I'm not here to answer a measure theory question.
I'm just pointing out an article that goes againsts yours.
Oh this isn't theory. Let's try again: Is there a limit to how much you can compress data?
I get why you don't _want_ to answer, since the answer proves that with the laws of physics in this universe LLMs don't store copies of their training data.
-
Oh this isn't theory. Let's try again: Is there a limit to how much you can compress data?
I get why you don't _want_ to answer, since the answer proves that with the laws of physics in this universe LLMs don't store copies of their training data.
> Oh this isn't theory. Let's try again: Is there a limit to how much you can compress data?
Well, I'm going to say that it depends on the domain here but you are probably after something specific here.
> I get why you don't _want_ to answer, since the answer proves that with the laws of physics in this universe LLMs don't store copies of their training data.
Oh, please do tell!
-
> Oh this isn't theory. Let's try again: Is there a limit to how much you can compress data?
Well, I'm going to say that it depends on the domain here but you are probably after something specific here.
> I get why you don't _want_ to answer, since the answer proves that with the laws of physics in this universe LLMs don't store copies of their training data.
Oh, please do tell!
No, it doesn't "depends".
https://en.wikipedia.org/wiki/Shannon%27s_source_coding_theorem
If LLMs stored their training data we would apparently be able to compress all of human knowledge into files easily downloadable onto regular computers, since that's the size of LLM models.
They don't. They do however learn and have better memories than human brains so they can indeed regurgitate 460 words in a row (that's from the paper you linked) from a source in some cases.
If you want to play debate, try learning the subject matter first.
-
No, it doesn't "depends".
https://en.wikipedia.org/wiki/Shannon%27s_source_coding_theorem
If LLMs stored their training data we would apparently be able to compress all of human knowledge into files easily downloadable onto regular computers, since that's the size of LLM models.
They don't. They do however learn and have better memories than human brains so they can indeed regurgitate 460 words in a row (that's from the paper you linked) from a source in some cases.
If you want to play debate, try learning the subject matter first.
> No, it doesn't "depends".
Oh okay! I guess, it was silly of myself to assume some constraints. However, I guess you win this one!
> If LLMs stored their training data we would apparently be able to compress all of human knowledge into files easily downloadable onto regular computers, since that's the size of LLM models.
Oh okay! So... LLMs don't store them at all?
> They don't. They do however learn and have better memories than human brains so they can indeed regurgitate 460 words in a row (that's from the paper you linked) from a source in some cases.
But now you are saying they do? If they could regurgitate 460 words in a row, sounds like they stored it or have some kind of memory of it right?
Like, that article was stating "reproduce up to 85-90% of held-out copyrighted books"
So... there is some representation in which one could consider it looking like compression in which HEY, look at that:
"A review of state-of-the-art techniques for large language model compression"https://link.springer.com/article/10.1007/s40747-025-02019-z
-
No, it doesn't "depends".
https://en.wikipedia.org/wiki/Shannon%27s_source_coding_theorem
If LLMs stored their training data we would apparently be able to compress all of human knowledge into files easily downloadable onto regular computers, since that's the size of LLM models.
They don't. They do however learn and have better memories than human brains so they can indeed regurgitate 460 words in a row (that's from the paper you linked) from a source in some cases.
If you want to play debate, try learning the subject matter first.
@troed good, what do you know about modern neuroscience? Because you know what they say: extraordinary claims require extraordinary proof. And you claimed that LLMs memorize things like the human brain, can you prove it? Because @ahto provided one of several.peer reviewed articles that prove without question that LLMs store high-probability training data essentially verbatim. But you didn't provide proof that the human brain store sparse matrixes and multiplies them.
-
@dome @0x0961h @jargoggles @eniko Did you just post text an LLM gave you? Pull your head out of your ass and type your own response.
-
@eniko AI enjoyers always react very emotional to criticism
-
-
@troed not the same thing and you know it. People looking at things and storing copies of someone else's potentially copyrighted data for training are two completely different things. Is it so hard to admit that there are externalities and they are bad no matter how you slice it?
Mistral AI is your run-of-the-mill AI company that does not disclose what's in their training sets, just like everybody else: https://help.mistral.ai/en/articles/347390-does-mistral-disclose-its-training-datasets
AI is being treated like that movie "Blood Diamond". I believe many are unaware, due to the loss of transparency, the costs involved to pull off that simple free prompt generation, which creators were stolen from, which towns had their houses demolished, or what resources were extracted for AI infrastructure.
Quili.ai made an ethical point sitting in as a team of human AI prompt responders to save their town of water scarcity; revealing the community we have forgotten.
-
> No, it doesn't "depends".
Oh okay! I guess, it was silly of myself to assume some constraints. However, I guess you win this one!
> If LLMs stored their training data we would apparently be able to compress all of human knowledge into files easily downloadable onto regular computers, since that's the size of LLM models.
Oh okay! So... LLMs don't store them at all?
> They don't. They do however learn and have better memories than human brains so they can indeed regurgitate 460 words in a row (that's from the paper you linked) from a source in some cases.
But now you are saying they do? If they could regurgitate 460 words in a row, sounds like they stored it or have some kind of memory of it right?
Like, that article was stating "reproduce up to 85-90% of held-out copyrighted books"
So... there is some representation in which one could consider it looking like compression in which HEY, look at that:
"A review of state-of-the-art techniques for large language model compression"https://link.springer.com/article/10.1007/s40747-025-02019-z
You're trying to argue against what amounts to natural laws, from your own lack of knowledge about the area.
That's the same thing as climate deniers do.
-
@troed good, what do you know about modern neuroscience? Because you know what they say: extraordinary claims require extraordinary proof. And you claimed that LLMs memorize things like the human brain, can you prove it? Because @ahto provided one of several.peer reviewed articles that prove without question that LLMs store high-probability training data essentially verbatim. But you didn't provide proof that the human brain store sparse matrixes and multiplies them.
I recommend Susan Blackmore's "Consciousness: An Introduction" and Douglas Hofstadters "I Am a Strange Loop" if you want more insight into moden neuroscience.
-
You're trying to argue against what amounts to natural laws, from your own lack of knowledge about the area.
That's the same thing as climate deniers do.
@troed @ahto I guess you mean scientific laws because natural laws are a philosophical concept. I also suppose you meant climate *change* deniers which - besides being uncalled for name-calling - is a bit ironic given that worsening climate change is indeed one of the externalities. Anyway back to the case in point it's not about a handful of words, we now have plenty of literature proving that training data contains verbatim copies of high-probability inputs regardless of what Shannon's theorem says, e.g.:
https://arxiv.org/pdf/2601.02671v1
> For Claude 3.7 Sonnet, we were able to extract four whole books near-verbatim, including two books under copyright in the U.S.: Harry Potter and the Sorcerer’s Stone and 1984
Anyway this brings me to another question. Why defending these systems in the face of the damage they do? Because you used them to write some software? That's your *expertise* that you used. That enabled you to do it, not the tool. That's the lie at the heart of these systems.
-
I recommend Susan Blackmore's "Consciousness: An Introduction" and Douglas Hofstadters "I Am a Strange Loop" if you want more insight into moden neuroscience.
-
@troed @ahto I guess you mean scientific laws because natural laws are a philosophical concept. I also suppose you meant climate *change* deniers which - besides being uncalled for name-calling - is a bit ironic given that worsening climate change is indeed one of the externalities. Anyway back to the case in point it's not about a handful of words, we now have plenty of literature proving that training data contains verbatim copies of high-probability inputs regardless of what Shannon's theorem says, e.g.:
https://arxiv.org/pdf/2601.02671v1
> For Claude 3.7 Sonnet, we were able to extract four whole books near-verbatim, including two books under copyright in the U.S.: Harry Potter and the Sorcerer’s Stone and 1984
Anyway this brings me to another question. Why defending these systems in the face of the damage they do? Because you used them to write some software? That's your *expertise* that you used. That enabled you to do it, not the tool. That's the lie at the heart of these systems.
@gabrielesvelto "high-probability inputs" + "near verbatim" is the same as what humans can do (ask your nearest Quran-reciter).
"regardless of what Shannon's theorem says"
Yeah that's when it just becomes funny.
"You claimed that LLMs work like brains"
No, I wrote: "What happens in their neural networks is very similar to what happens in a human brain when learning"
Do you dispute that statement? What is it the whole field of neural networks is modelled after?
-
@gabrielesvelto "high-probability inputs" + "near verbatim" is the same as what humans can do (ask your nearest Quran-reciter).
"regardless of what Shannon's theorem says"
Yeah that's when it just becomes funny.
"You claimed that LLMs work like brains"
No, I wrote: "What happens in their neural networks is very similar to what happens in a human brain when learning"
Do you dispute that statement? What is it the whole field of neural networks is modelled after?
@troed @ahto what humans can do and how information is stored are two different things and you're deliberately mixing up the concepts. Shannon's theorem is *specifically* about storage, so how do you want to slice this particular piece of bread?
That aside let's go back to your original extraordinary claim that neural networks learn like human brains: where's your proof? Your peer-reviewed proof complete with reproducible results? Because they really are nothing alike.
https://www.quantamagazine.org/ai-is-nothing-like-a-brain-and-thats-ok-20250430/
> But they’re not the same — and probably never will be. “[Neural networks] are now sufficiently different from the way that actual brains are in so many different ways that I think it’s actually more sensible to think of them as a really different information-processing object,” said Shine, the systems neurobiologist, “one that’s extremely interesting in its own right.”
-
@troed @ahto what humans can do and how information is stored are two different things and you're deliberately mixing up the concepts. Shannon's theorem is *specifically* about storage, so how do you want to slice this particular piece of bread?
That aside let's go back to your original extraordinary claim that neural networks learn like human brains: where's your proof? Your peer-reviewed proof complete with reproducible results? Because they really are nothing alike.
https://www.quantamagazine.org/ai-is-nothing-like-a-brain-and-thats-ok-20250430/
> But they’re not the same — and probably never will be. “[Neural networks] are now sufficiently different from the way that actual brains are in so many different ways that I think it’s actually more sensible to think of them as a really different information-processing object,” said Shine, the systems neurobiologist, “one that’s extremely interesting in its own right.”
"Shannon's theorem is *specifically* about storage"
Yes, that's the fact that disproves LLMs storing their training data verbatim. This is how you make a logic proof.
"let's go back to your original extraordinary claim that neural networks learn like human brains: where's your proof? Your peer-reviewed proof complete with reproducible results? Because they really are nothing alike."
Here's Google's paper from 2017 that is behind all the LLM architectures today:
https://arxiv.org/abs/1706.03762
Here's an explanation to how that's similar to human brains:
I wrote my first back-propagating neural network in 1993. You?
-
You're trying to argue against what amounts to natural laws, from your own lack of knowledge about the area.
That's the same thing as climate deniers do.
I asked about the domain, we could be talking about infinite sets, finite sets, where the alphabet is infinite, lossy and lossless and other kind of nuances here. Could have been asking about quantum mechanics here and error correction.
You got specific and attempted to portray a belief I never stated like so as well.
I was more open to things I may not have considered and read up on them. You wanted to try and flex, okay, that's fine.
> I get why you don't _want_ to answer, since the answer proves that with the laws of physics in this universe LLMs don't store copies of their training data.
Which is you speculating about things without evidence here.
To be pedantic here as well:
> Oh this isn't theory
I mean, I was off with measure theory but I guess it belongs to information theory here so not true.
> You're trying to argue against what amounts to natural laws, from your own lack of knowledge about the area.
No, I just outlined the works and I had cited work by academics who have done research in this area and have shown claims that it is able to be reconstructed and leading to the conclusion that they are encoding the data in some form.
-
I asked about the domain, we could be talking about infinite sets, finite sets, where the alphabet is infinite, lossy and lossless and other kind of nuances here. Could have been asking about quantum mechanics here and error correction.
You got specific and attempted to portray a belief I never stated like so as well.
I was more open to things I may not have considered and read up on them. You wanted to try and flex, okay, that's fine.
> I get why you don't _want_ to answer, since the answer proves that with the laws of physics in this universe LLMs don't store copies of their training data.
Which is you speculating about things without evidence here.
To be pedantic here as well:
> Oh this isn't theory
I mean, I was off with measure theory but I guess it belongs to information theory here so not true.
> You're trying to argue against what amounts to natural laws, from your own lack of knowledge about the area.
No, I just outlined the works and I had cited work by academics who have done research in this area and have shown claims that it is able to be reconstructed and leading to the conclusion that they are encoding the data in some form.
@ahto "they are encoding the data in some form."
Yes, they're creating lossy memories from the ingested training. That's what human brains also do.
Neither of us "store copies".
-
"Shannon's theorem is *specifically* about storage"
Yes, that's the fact that disproves LLMs storing their training data verbatim. This is how you make a logic proof.
"let's go back to your original extraordinary claim that neural networks learn like human brains: where's your proof? Your peer-reviewed proof complete with reproducible results? Because they really are nothing alike."
Here's Google's paper from 2017 that is behind all the LLM architectures today:
https://arxiv.org/abs/1706.03762
Here's an explanation to how that's similar to human brains:
I wrote my first back-propagating neural network in 1993. You?
> Yes, that's the fact that disproves LLMs storing their training data verbatim. This is how you make a logic proof.
Really? Here's a type of lossless storage that respects Shannon's theorem while storing verbatim data: I scrape X data off the internet, I can keep only Y data where Y < X. I throw away all non-copyrighted material until I get only Y data. Shannon's theorem was respected and I have verbatim data that I am now distributing without consent, attribution or compensation.
> https://
arxiv.org/abs/1706.03762This contains exactly zero proof that LLMs and brains operate alike.
← where I wish sloperators were put