LLM advocates still don’t seem to be able to comprehend that ordering the machine not to ‘make stuff up’ doesn’t help.
-
LLM are a Skinner Box.
@Aedius @skotchygut @benjamineskola
Designed by wannabe supervillains, built with wartime-scale resources.
The hell do people expect??
-
@nelson @benjamineskola @solonovamax
yeah, i think my take from about a year ago still mostly holds up https://biplus.social/@linkplay/114828181247605258@linkplay @nelson @benjamineskola @solonovamax
It's just rolling linguistic dice, words bouncing around between probablistic paddles in a bigass pachinko matrix. It's not designed to vet facts. It's designed to regurgitate plausible spitwads that RESEMBLE facts. And the weights behind all those paddles and slots are tuned to whatever agenda the designers wish.
...And the designers serve planetwrecking technofascist war profiteers who party with people like Epstein.
Why would anyone ever trust it with so much as a goddamn casserole recipe??
-
@Su_G @Aedius @benjamineskola I shamelessly stole the term from @davidgerard
-
LLM advocates still don’t seem to be able to comprehend that ordering the machine not to ‘make stuff up’ doesn’t help. It doesn’t know when it’s making stuff up, and it couldn’t change that even if you told it to. (In fact it’s always just making stuff up, and is only ever true by chance.)
Part of why I’m so negative about them is that their advocates simply do not understand how they work and do not seem to want to.
@benjamineskola
Even if you curate the source text in detail, there is no guarantee that you avoid falsehoods and imaginings. When you just feed the entire WWW blindly into your LLM, of course all bets are off. -
@benjamineskola
Even if you curate the source text in detail, there is no guarantee that you avoid falsehoods and imaginings. When you just feed the entire WWW blindly into your LLM, of course all bets are off.@mrundkvist yes exactly. It’s a complete misunderstanding of how these tools work.
-
@mrundkvist yes exactly. It’s a complete misunderstanding of how these tools work.
@benjamineskola
Also a breathtakingly naïve idea about truth! -
LLM advocates still don’t seem to be able to comprehend that ordering the machine not to ‘make stuff up’ doesn’t help. It doesn’t know when it’s making stuff up, and it couldn’t change that even if you told it to. (In fact it’s always just making stuff up, and is only ever true by chance.)
Part of why I’m so negative about them is that their advocates simply do not understand how they work and do not seem to want to.
@benjamineskola Part of why I'm so frustrated is that their detractors simply do not understand how they work and do not seem to want to.
(No, they're not simply correct "by chance")
/actual very very senior greybeard dev who challenged his own erroneous convictions regarding LLMs and now find them useful
-
@benjamineskola Part of why I'm so frustrated is that their detractors simply do not understand how they work and do not seem to want to.
(No, they're not simply correct "by chance")
/actual very very senior greybeard dev who challenged his own erroneous convictions regarding LLMs and now find them useful
@troed whatever you feel the need to tell yourself to justify it.
-
@troed whatever you feel the need to tell yourself to justify it.
@benjamineskola Do you ever ponder that you might be wrong about something?
-
@benjamineskola Do you ever ponder that you might be wrong about something?
@troed all the time! But that doesn’t mean I’m wrong here.
-
@troed all the time! But that doesn’t mean I’m wrong here.
@benjamineskola You are. Interested in learning?
(I was wrong about LLMs for development, took the time to learn, and changed my mind)
-
@benjamineskola You are. Interested in learning?
(I was wrong about LLMs for development, took the time to learn, and changed my mind)
@troed I’m not really interested in hearing your justifications for why actually it makes total sense to just tell the text generator ‘don’t make stuff up!’ as if it’s doing so by choice.
-
@troed I’m not really interested in hearing your justifications for why actually it makes total sense to just tell the text generator ‘don’t make stuff up!’ as if it’s doing so by choice.
@benjamineskola You do you.
(I didn't say anything about "don't make stuff up" - that's a strawman argument you successfully fought)
-
@benjamineskola You do you.
(I didn't say anything about "don't make stuff up" - that's a strawman argument you successfully fought)
@troed It’s clearly not a strawman argument, because it’s a common problem that I gave a real-world example of.
And if that’s not what you want to discuss I don’t understand why you decided to start an argument about it.
-
@troed It’s clearly not a strawman argument, because it’s a common problem that I gave a real-world example of.
And if that’s not what you want to discuss I don’t understand why you decided to start an argument about it.
@benjamineskola I replied specifically about your claim that LLMs are only correct "by chance". That's not how LLMs work.
I'm not responsible for other statements made by other people.
-
@benjamineskola I replied specifically about your claim that LLMs are only correct "by chance". That's not how LLMs work.
I'm not responsible for other statements made by other people.
@troed LLMs do not have a concept of truth or falsehood. The likelihood of a particular output is based on its representation in the input data and the process of training, not on its truth value. Prompting it to “never make stuff up” and similar is a sign of misunderstanding, because LLMs do not have a concept of “making stuff up” — all outputs are produced in the same way regardless of truth or falsehood.
It may be correct to say that a ‘true’ output is more probable because it’s more common in the training data or has been reinforced by the training process, etc; but this is not the same as an LLM being able to distinguish between truth and falsehood (and anyway is not reliably correct). The likelihood of an output is independent of its truth-value and as such one is relying on the probability of it producing a true/useful output.
-
@troed LLMs do not have a concept of truth or falsehood. The likelihood of a particular output is based on its representation in the input data and the process of training, not on its truth value. Prompting it to “never make stuff up” and similar is a sign of misunderstanding, because LLMs do not have a concept of “making stuff up” — all outputs are produced in the same way regardless of truth or falsehood.
It may be correct to say that a ‘true’ output is more probable because it’s more common in the training data or has been reinforced by the training process, etc; but this is not the same as an LLM being able to distinguish between truth and falsehood (and anyway is not reliably correct). The likelihood of an output is independent of its truth-value and as such one is relying on the probability of it producing a true/useful output.
@benjamineskola Correct - for coding and system maintenance tasks that can be evaluated as correct/not correct from their functionality an LLM will gravitate towards the correct solution based on how that's much more common in the training data.
https://knowprose.com/2026/03/why-ai-writing-converges-the-adjacent-possible-of-language-models/
This makes LLMs very useful in software development. I have no one telling what to use or not use - yet I use them because doing so has measurably increased my productivity - especially in areas outside of my own core competence.
-
@benjamineskola Correct - for coding and system maintenance tasks that can be evaluated as correct/not correct from their functionality an LLM will gravitate towards the correct solution based on how that's much more common in the training data.
https://knowprose.com/2026/03/why-ai-writing-converges-the-adjacent-possible-of-language-models/
This makes LLMs very useful in software development. I have no one telling what to use or not use - yet I use them because doing so has measurably increased my productivity - especially in areas outside of my own core competence.
@troed Okay, so we agree that I’m correct; I still don’t understand what you felt the need to argue about.
-
LLM advocates still don’t seem to be able to comprehend that ordering the machine not to ‘make stuff up’ doesn’t help. It doesn’t know when it’s making stuff up, and it couldn’t change that even if you told it to. (In fact it’s always just making stuff up, and is only ever true by chance.)
Part of why I’m so negative about them is that their advocates simply do not understand how they work and do not seem to want to.
Well, you could ask the AI your question, then check all the claims, respond with your findings, check the new claims, etc. until everything is OK.
But all that checking would be work. You can't be lazier, than without the AI.
Of course it helps, when you know the topic yourself. Then the checking-part is a lot easier.
But learning that would be work. You can't be lazier, than without the AI.
-
J jwcph@helvede.net shared this topic
