This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall I've been saying a version of this for years (though much less eloquently). It also applies to technical writing in any field.
On LLMs' having intent - yes, they don't, but arguably they always produce bullshit: https://link.springer.com/article/10.1007/s10676-024-09775-5
-
@david_chisnall Sorry, but this article is not a good source. There is nothing stated about the bug(s) and the narrative is the dev has brought 2 bugs to prod in 6 months and was threatend to be fired. That alone does not sound plausible. The rest is the usual fear mongering. As if subtile bugs is something new.
@demiurg @david_chisnall Production being down can lose you millions. I know someone who caused that (before slop generators) but suffered no consequences.
-
@thirstybear @david_chisnall pretty sure your style is better than this.
@hbons @thirstybear @david_chisnall Just in case introduce some spelling errors.
-
I'm no expert but my naive under is that theorem provers are designed to be correct...
.. and some people are using Llama to "search" for candidate proofs which are only given credibility once they pass a theorem provers check.
In this sense the output of an llm is filtered by a "correctness filter".
I may not like this workflow but if it advances human knowledge then there may be a case for it, subject to environmental and other ethical concerns.
Did I misunderstand your observation?
@rzeta0 @bms @david_chisnall That was probably a reference to the recent incident in which a LLM exploited a bug in the theorem prover to make it accept an invalid proof as valid.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall “LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.”
LLMs optimize for verisimilitude, not veracity.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall funny that this senior developer suggests tests are one of the only things ok to write with LLMs, since those are the one thing I would *never* entrust to a probabilistic machine. With correctly constructed tests, I would at least hypothetically know the slop code’s output is correct, no matter how bad the runtime is.
-
@david_chisnall “LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.”
LLMs optimize for verisimilitude, not veracity.
Oooo, •nice• (& subtle!) distinction!
-
I'm no expert but my naive under is that theorem provers are designed to be correct...
.. and some people are using Llama to "search" for candidate proofs which are only given credibility once they pass a theorem provers check.
In this sense the output of an llm is filtered by a "correctness filter".
I may not like this workflow but if it advances human knowledge then there may be a case for it, subject to environmental and other ethical concerns.
Did I misunderstand your observation?
As @pluralistic has frequently pointed out, LLMs are not without good uses. But like any hammer, you maybe don't want to be using them on •everything•, which seems to be the attitude held by the cult-members.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
—Cal Newport—!
-
Yes.
There are several possibilities for that somewhere (including programming language design) and they don't have to be mutually exclusive. -
@david_chisnall Great article. I want to zoom in on the last thing he says:
‘AI is not a magic “infinity machine” that can solve all our problems… It’s a normal technology, and perhaps it’s time we start talking about it that way.’I assert that for A LOT of people, they already viewed most tech as magic even before LLMs went big. I think a lot of people ARE treating LLMs like they treat normal tech. Though when the author says it’s time WE talk about LLMs as normal tech, maybe he’s thinking of tech people and programmers as the WE, not people in general.
I think a lot of tech had already exceeded the normal person’s ability to understand and predict. So this is just really bad coincident timing for LLMs to appear. I have hope that the scales will fall off programmers’ eyes. I don’t have much hope for everyone else.
...depending, of course, on your threshold for "normal person." I've encountered far more people that I would like who are foggy on the distinctions between "network," "computer," "operating system," "software (application)," & "document," & as a consequence get lost very quickly when trying to understand what a computer can accomplish. Likewise the difference between "The Internet" & "Facebook." (Though I encounter that conflation more rarely. Thank Ghu.)
-
@paco @david_chisnall And then there's the problem that a lot of the general audience will assume "oh, if it had such glaring issues then the experts would not be using it or warn us about it" ... and, well ...
I suspect this is of a piece with, "that candidate is so obviously evil, nobody could possibly vote for them...."
Which, um.
-
@TauPan I cringe so much when I see people say 'I use it for repetitive things, like tests!'. Because writing good tests requires understanding which bits in the code are corner cases and writing tests that exercise those. I've seen LLM-generated tests with hundreds of tests for the happy path and none that trigger any of the error-handling.
I can maybe imagine a loop with an LLM and a coverage tool to get proper coverage, but then you'd need to review the tests generated by the LLMs for the corner cases to make sure you weren't just generating tests that enforce bugs in the implementation.
...like 3-dimensional survivor bias....
-
@ChemicalEyeGuy @david_chisnall
"Maybe it's more efficient just to burn it all down & start from scratch?"
"A case could be made...."
Oh wait—that was DOGE's rationale, too....
-
@david_chisnall That's (on of) my concern(s) with LLM submissions too.
It's going to be so much more work to review them than reviewing human submissions, and at the same time the payoff is less valuable. Human submissions, even if they're flawed at first, may result in a new contributor. Reviewing LLM submissions will not grow a new contributor.I think one of the major unintentional lessons of this era (both technically & politically) is the value of deep human experience.
Edit: Value & intrinsically un-automatability.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall Well, I dunno... I've been using Claude since the beginning of the year and it *is* amazing. It has increased my productivity tremendously. Of course, I never trust it blindly and always verify by hand what it generates. And, yes, I sometimes find bugs in the generated code. Nevertheless, and despite the time I have to spend checking its generated code, it is still an amazing productivity-increasing tool.
-
@david_chisnall Well, I dunno... I've been using Claude since the beginning of the year and it *is* amazing. It has increased my productivity tremendously. Of course, I never trust it blindly and always verify by hand what it generates. And, yes, I sometimes find bugs in the generated code. Nevertheless, and despite the time I have to spend checking its generated code, it is still an amazing productivity-increasing tool.
@bontchev @david_chisnall OP makes a really subtle point there that I think deserves some more reflection; I'm going to be thinking about that angle for a while.
-
@david_chisnall do you often brine your supply chain attacks in house? i normally outsource it.
@dysfun @david_chisnall This is a very salty take.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall @briankrebs I also often point out that, with a human colleague, I know what they are good or bad at, I know how they are feeling today, I know how good they understand the context of the task and therefore I know where to look for bugs. LLM-generated code: no idea, just as you wrote.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
You forgot the 'pleasing' part.
A "statistically plausible pleasing machine." LLMs will give you the answer you want based on your repeated clarification prompts.
