This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been.
-
It’s something you can predict if you’ve read LLM-generated prose too. They generate entirely plausible-sounding text. Spotting the errors is hard. If you’d written the text, you wouldn’t introduce errors from completely fabricated data and citations. If someone who wasn’t actively trying to trick you wrote the text, they wouldn’t include made-up numbers and citations. But somehow people expect code to be magical and special.
@david_chisnall I once told ChatGPT to create a non-standard x-to-y belt-balancer for the game factorio AND print out the encoded export-string for me so I can easily import it into the game. It was a test, not a real thing I needed. I just wanted to know what that thing puts out if anything at all. And it did put out something that looked very, very plausible. Even said string. Well, guess what happened when I tried to import it into the game? It was invalid of course!
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall looks like slop writing though.
![😕](https://forum.fedi.dk/assets/plugins/nodebb-plugin-emoji/emoji/android/1f615.png?v=7979fdcf9c7 ":\")
"But what if the reality here is more complicated?"
"This is a key point."
"Here’s the thing: he’s not alone."
"But it does emphasize an important truth:" -
@david_chisnall I once told ChatGPT to create a non-standard x-to-y belt-balancer for the game factorio AND print out the encoded export-string for me so I can easily import it into the game. It was a test, not a real thing I needed. I just wanted to know what that thing puts out if anything at all. And it did put out something that looked very, very plausible. Even said string. Well, guess what happened when I tried to import it into the game? It was invalid of course!
@spitfire I think that’s part of why people trust it for code. You know it was invalid because the parser rejected it. If you have to compile the code, run it, and pass tests, then you have some confidence that it is not completely wrong. The problem is that you have a tool that is especially good at producing subtly wrong output.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall this piece would be less annoying if it didn't offer the evidence against Ai coding and then say it was good really
-
@david_chisnall looks like slop writing though.
"But what if the reality here is more complicated?"
"This is a key point."
"Here’s the thing: he’s not alone."
"But it does emphasize an important truth:"@hbons @david_chisnall yeah that's the other problem
-
@david_chisnall this piece would be less annoying if it didn't offer the evidence against Ai coding and then say it was good really
@davidgerard it's always hard to know when people are hedging because they don't want to hurt people's feelings, or if they've just adopted magical thinking after staring into Pandora's Box @david_chisnall
-
@david_chisnall this piece would be less annoying if it didn't offer the evidence against Ai coding and then say it was good really
@davidgerard @david_chisnall see the other popular article about the decline in online writing because everyone uses AI which ends with “but my use of AI is good, actually”
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall “LLMs, by their nature, generate statistically plausible output." and “trained on code that made it past code review”.
<Ding!>
Bang on target. -
@david_chisnall looks like slop writing though.
"But what if the reality here is more complicated?"
"This is a key point."
"Here’s the thing: he’s not alone."
"But it does emphasize an important truth:"@hbons @david_chisnall All of those phrases were in common use before LLMs. That's why LLMs use them. You can't conclude from their presence that this is slop.
-
@david_chisnall looks like slop writing though.
"But what if the reality here is more complicated?"
"This is a key point."
"Here’s the thing: he’s not alone."
"But it does emphasize an important truth:"@hbons @david_chisnall Damn…that's my (non-AI) writing style screwed then
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall That's (on of) my concern(s) with LLM submissions too.
It's going to be so much more work to review them than reviewing human submissions, and at the same time the payoff is less valuable. Human submissions, even if they're flawed at first, may result in a new contributor. Reviewing LLM submissions will not grow a new contributor. -
@hbons @david_chisnall All of those phrases were in common use before LLMs. That's why LLMs use them. You can't conclude from their presence that this is slop.
@cholling @david_chisnall from this amount I’m 90% certain.
-
@hbons @david_chisnall Damn…that's my (non-AI) writing style screwed then
@thirstybear @david_chisnall pretty sure your style is better than this.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall True. I've found that strict TDD is the best way to handle it. The Superpowers plugin is excellent at enforcing it with the agents.
I'm seeing remarkably few subtle bugs in agent produced code.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
> I’m only being slightly flippant when I say LLMs are a mechanism for brining supply chain attacks in house.
I am, however, absolutely serious when I say that outsourcing code production to a handful providers is the series of supply chain attacks in the making.
The use of LLMs is a cherry on top.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall exactly the argument i keep making and people keep telling me I have to be wrong.
It gets tiring.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall And kids, don't use LLMs for test code either! Subtle bugs in test code are just as hard to catch as in production code and that compromises your production code. Heck, don't use that shit for anything at all.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall Great article. I want to zoom in on the last thing he says:
‘AI is not a magic “infinity machine” that can solve all our problems… It’s a normal technology, and perhaps it’s time we start talking about it that way.’I assert that for A LOT of people, they already viewed most tech as magic even before LLMs went big. I think a lot of people ARE treating LLMs like they treat normal tech. Though when the author says it’s time WE talk about LLMs as normal tech, maybe he’s thinking of tech people and programmers as the WE, not people in general.
I think a lot of tech had already exceeded the normal person’s ability to understand and predict. So this is just really bad coincident timing for LLMs to appear. I have hope that the scales will fall off programmers’ eyes. I don’t have much hope for everyone else.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
You make a very key point, which is the one I try to convey to non-technical people I speak to about the risks of this stuff.
Some systems are trained to be correct. That is the training objective.
Some systems are trained to look correct, to get past a forgery spotter. That isn't the same as being trained to be correct.
This is a subtle (or not!) difference between say an LLM and post code hand writing classifier in the mail system.
This does "click" with most of the people I try to convey this diifference to.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall Studies show that using an LLM to outsource your thinking degrades your own performance on that same topic. Reading and analyzing code is also harder that writing it. Put these two together and it becomes difficult for people to review this generated code and spot any subtle bugs. Particularly since the really nasty ones tend to not be within any given section but with how it interacts with other modules.
