This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall True. I've found that strict TDD is the best way to handle it. The Superpowers plugin is excellent at enforcing it with the agents.
I'm seeing remarkably few subtle bugs in agent produced code.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
> I’m only being slightly flippant when I say LLMs are a mechanism for brining supply chain attacks in house.
I am, however, absolutely serious when I say that outsourcing code production to a handful providers is the series of supply chain attacks in the making.
The use of LLMs is a cherry on top.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall exactly the argument i keep making and people keep telling me I have to be wrong.
It gets tiring.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall And kids, don't use LLMs for test code either! Subtle bugs in test code are just as hard to catch as in production code and that compromises your production code. Heck, don't use that shit for anything at all.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall Great article. I want to zoom in on the last thing he says:
‘AI is not a magic “infinity machine” that can solve all our problems… It’s a normal technology, and perhaps it’s time we start talking about it that way.’I assert that for A LOT of people, they already viewed most tech as magic even before LLMs went big. I think a lot of people ARE treating LLMs like they treat normal tech. Though when the author says it’s time WE talk about LLMs as normal tech, maybe he’s thinking of tech people and programmers as the WE, not people in general.
I think a lot of tech had already exceeded the normal person’s ability to understand and predict. So this is just really bad coincident timing for LLMs to appear. I have hope that the scales will fall off programmers’ eyes. I don’t have much hope for everyone else.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
You make a very key point, which is the one I try to convey to non-technical people I speak to about the risks of this stuff.
Some systems are trained to be correct. That is the training objective.
Some systems are trained to look correct, to get past a forgery spotter. That isn't the same as being trained to be correct.
This is a subtle (or not!) difference between say an LLM and post code hand writing classifier in the mail system.
This does "click" with most of the people I try to convey this diifference to.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall Studies show that using an LLM to outsource your thinking degrades your own performance on that same topic. Reading and analyzing code is also harder that writing it. Put these two together and it becomes difficult for people to review this generated code and spot any subtle bugs. Particularly since the really nasty ones tend to not be within any given section but with how it interacts with other modules.
-
@amenonsen Sounds terrifying. I’ve seen ‘AI’ reviews and they have a huge number of false positives. Actually applying their suggestions makes the code worse on average. Reviewing the output from ‘AI’ reviews takes a lot of effort.
But, beyond that, it’s missing two of the key values of code review: ensuring that the code is understood by at least two people and ensuring that the code is understandable by someone other than the author. If you give up on that, you are assuming that code that makes it into your tree will be understood by at most one person, who may leave the project at any point.
I’ve worked on projects like that long before LLMs were a thing and, well, there are good reasons sensible projects try really hard to avoid becoming like that again.
LLMs write pretty good spec. If you don't like its code, fine, I can understand that sort of thing.
Beginners ask for code. Experienced people ask for spec. That's the TL;DR
-
LLMs write pretty good spec. If you don't like its code, fine, I can understand that sort of thing.
Beginners ask for code. Experienced people ask for spec. That's the TL;DR
As for code review, let me tell you, Opus with a million tokens on the job Impressed Me Deeply.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall
It makes me think of The Underhanded C Contest. -
@david_chisnall
It makes me think of The Underhanded C Contest.@kirtai Exactly. Imagine doing code review where every PR was an entry into the underhanded C contest.
-
@david_chisnall That's (on of) my concern(s) with LLM submissions too.
It's going to be so much more work to review them than reviewing human submissions, and at the same time the payoff is less valuable. Human submissions, even if they're flawed at first, may result in a new contributor. Reviewing LLM submissions will not grow a new contributor.@kp @david_chisnall same riff here https://1password.com/blog/why-ai-generated-patches-still-require-human-review slop will doom us
-
@david_chisnall And kids, don't use LLMs for test code either! Subtle bugs in test code are just as hard to catch as in production code and that compromises your production code. Heck, don't use that shit for anything at all.
@TauPan @david_chisnall harder to find, because of lack of intent.
-
@david_chisnall And kids, don't use LLMs for test code either! Subtle bugs in test code are just as hard to catch as in production code and that compromises your production code. Heck, don't use that shit for anything at all.
@TauPan I cringe so much when I see people say 'I use it for repetitive things, like tests!'. Because writing good tests requires understanding which bits in the code are corner cases and writing tests that exercise those. I've seen LLM-generated tests with hundreds of tests for the happy path and none that trigger any of the error-handling.
I can maybe imagine a loop with an LLM and a coverage tool to get proper coverage, but then you'd need to review the tests generated by the LLMs for the corner cases to make sure you weren't just generating tests that enforce bugs in the implementation.
-
@david_chisnall “LLMs, by their nature, generate statistically plausible output." and “trained on code that made it past code review”.
<Ding!>
Bang on target.@thirstybear @david_chisnall except it is also trained on tons of shitposts and repos that don’t do code review
-
@david_chisnall Great article. I want to zoom in on the last thing he says:
‘AI is not a magic “infinity machine” that can solve all our problems… It’s a normal technology, and perhaps it’s time we start talking about it that way.’I assert that for A LOT of people, they already viewed most tech as magic even before LLMs went big. I think a lot of people ARE treating LLMs like they treat normal tech. Though when the author says it’s time WE talk about LLMs as normal tech, maybe he’s thinking of tech people and programmers as the WE, not people in general.
I think a lot of tech had already exceeded the normal person’s ability to understand and predict. So this is just really bad coincident timing for LLMs to appear. I have hope that the scales will fall off programmers’ eyes. I don’t have much hope for everyone else.
@paco @david_chisnall And then there's the problem that a lot of the general audience will assume "oh, if it had such glaring issues then the experts would not be using it or warn us about it" ... and, well ...
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall i am really getting sick of posts and articles blaming AI where actually the humans and managers are to blame.
-
You make a very key point, which is the one I try to convey to non-technical people I speak to about the risks of this stuff.
Some systems are trained to be correct. That is the training objective.
Some systems are trained to look correct, to get past a forgery spotter. That isn't the same as being trained to be correct.
This is a subtle (or not!) difference between say an LLM and post code hand writing classifier in the mail system.
This does "click" with most of the people I try to convey this diifference to.
@rzeta0 @david_chisnall Arguably this evasion of which you write has just happened with theorem provers.
-
@david_chisnall Great article. I want to zoom in on the last thing he says:
‘AI is not a magic “infinity machine” that can solve all our problems… It’s a normal technology, and perhaps it’s time we start talking about it that way.’I assert that for A LOT of people, they already viewed most tech as magic even before LLMs went big. I think a lot of people ARE treating LLMs like they treat normal tech. Though when the author says it’s time WE talk about LLMs as normal tech, maybe he’s thinking of tech people and programmers as the WE, not people in general.
I think a lot of tech had already exceeded the normal person’s ability to understand and predict. So this is just really bad coincident timing for LLMs to appear. I have hope that the scales will fall off programmers’ eyes. I don’t have much hope for everyone else.
@paco @david_chisnall This closely correlates with "software development proficiency is not necessarily correlated with proficiency in general systems theory, cognitive science, or pushing the limits of human consciousness itself", the latter of which is not carte blanche to treat Esalen Institute outputs uncritically
-
@ChemicalEyeGuy @david_chisnall I'm particularly fond of the term "cognitive knotweed" at the moment, though I usually tend to employ a C-word...
