This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall do you often brine your supply chain attacks in house? i normally outsource it.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall "it is a machine that is trained on code that made it past code review". Are you sure of that? Since a large part of the corpus is presumably code scraped from the likes of StackExchange and GitHub, I'd argue that it's mostly trained on newbies' misconceptions and unfinished personal projects which were abandoned because they were too ambitious or didn't work.
-
@david_chisnall do you often brine your supply chain attacks in house? i normally outsource it.
@dysfun Well, that used to be cheaper, but with subsidised LLM subscriptions it makes sense to onshore it.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall I was reading a thread recently in which people were trying to formulate an LLM policy for a project, and someone said (words to the effect that) we would ideally have both AI and human reviews, but if they had to get rid of one, they would keep the AI reviews.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall Sorry, but this article is not a good source. There is nothing stated about the bug(s) and the narrative is the dev has brought 2 bugs to prod in 6 months and was threatend to be fired. That alone does not sound plausible. The rest is the usual fear mongering. As if subtile bugs is something new.
-
@dysfun Well, that used to be cheaper, but with subsidised LLM subscriptions it makes sense to onshore it.
@david_chisnall you say this, but salt is cheap out east.
-
@david_chisnall I was reading a thread recently in which people were trying to formulate an LLM policy for a project, and someone said (words to the effect that) we would ideally have both AI and human reviews, but if they had to get rid of one, they would keep the AI reviews.
@amenonsen Sounds terrifying. I’ve seen ‘AI’ reviews and they have a huge number of false positives. Actually applying their suggestions makes the code worse on average. Reviewing the output from ‘AI’ reviews takes a lot of effort.
But, beyond that, it’s missing two of the key values of code review: ensuring that the code is understood by at least two people and ensuring that the code is understandable by someone other than the author. If you give up on that, you are assuming that code that makes it into your tree will be understood by at most one person, who may leave the project at any point.
I’ve worked on projects like that long before LLMs were a thing and, well, there are good reasons sensible projects try really hard to avoid becoming like that again.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall I'm telling people since day one that this might be a major problem down the road. But hey - I am not a developer myself, just an ordinary computer-toucher. What do I know?
-
@david_chisnall I'm telling people since day one that this might be a major problem down the road. But hey - I am not a developer myself, just an ordinary computer-toucher. What do I know?
It’s something you can predict if you’ve read LLM-generated prose too. They generate entirely plausible-sounding text. Spotting the errors is hard. If you’d written the text, you wouldn’t introduce errors from completely fabricated data and citations. If someone who wasn’t actively trying to trick you wrote the text, they wouldn’t include made-up numbers and citations. But somehow people expect code to be magical and special.
-
It’s something you can predict if you’ve read LLM-generated prose too. They generate entirely plausible-sounding text. Spotting the errors is hard. If you’d written the text, you wouldn’t introduce errors from completely fabricated data and citations. If someone who wasn’t actively trying to trick you wrote the text, they wouldn’t include made-up numbers and citations. But somehow people expect code to be magical and special.
@david_chisnall I once told ChatGPT to create a non-standard x-to-y belt-balancer for the game factorio AND print out the encoded export-string for me so I can easily import it into the game. It was a test, not a real thing I needed. I just wanted to know what that thing puts out if anything at all. And it did put out something that looked very, very plausible. Even said string. Well, guess what happened when I tried to import it into the game? It was invalid of course!
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall looks like slop writing though.
![😕](https://forum.fedi.dk/assets/plugins/nodebb-plugin-emoji/emoji/android/1f615.png?v=7979fdcf9c7 ":\")
"But what if the reality here is more complicated?"
"This is a key point."
"Here’s the thing: he’s not alone."
"But it does emphasize an important truth:" -
@david_chisnall I once told ChatGPT to create a non-standard x-to-y belt-balancer for the game factorio AND print out the encoded export-string for me so I can easily import it into the game. It was a test, not a real thing I needed. I just wanted to know what that thing puts out if anything at all. And it did put out something that looked very, very plausible. Even said string. Well, guess what happened when I tried to import it into the game? It was invalid of course!
@spitfire I think that’s part of why people trust it for code. You know it was invalid because the parser rejected it. If you have to compile the code, run it, and pass tests, then you have some confidence that it is not completely wrong. The problem is that you have a tool that is especially good at producing subtly wrong output.
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall this piece would be less annoying if it didn't offer the evidence against Ai coding and then say it was good really
-
@david_chisnall looks like slop writing though.
"But what if the reality here is more complicated?"
"This is a key point."
"Here’s the thing: he’s not alone."
"But it does emphasize an important truth:"@hbons @david_chisnall yeah that's the other problem
-
@david_chisnall this piece would be less annoying if it didn't offer the evidence against Ai coding and then say it was good really
@davidgerard it's always hard to know when people are hedging because they don't want to hurt people's feelings, or if they've just adopted magical thinking after staring into Pandora's Box @david_chisnall
-
@david_chisnall this piece would be less annoying if it didn't offer the evidence against Ai coding and then say it was good really
@davidgerard @david_chisnall see the other popular article about the decline in online writing because everyone uses AI which ends with “but my use of AI is good, actually”
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall “LLMs, by their nature, generate statistically plausible output." and “trained on code that made it past code review”.
<Ding!>
Bang on target. -
@david_chisnall looks like slop writing though.
"But what if the reality here is more complicated?"
"This is a key point."
"Here’s the thing: he’s not alone."
"But it does emphasize an important truth:"@hbons @david_chisnall All of those phrases were in common use before LLMs. That's why LLMs use them. You can't conclude from their presence that this is slop.
-
@david_chisnall looks like slop writing though.
"But what if the reality here is more complicated?"
"This is a key point."
"Here’s the thing: he’s not alone."
"But it does emphasize an important truth:"@hbons @david_chisnall Damn…that's my (non-AI) writing style screwed then
-
This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems
LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.
Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.
And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.
An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.
I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.
@david_chisnall That's (on of) my concern(s) with LLM submissions too.
It's going to be so much more work to review them than reviewing human submissions, and at the same time the payoff is less valuable. Human submissions, even if they're flawed at first, may result in a new contributor. Reviewing LLM submissions will not grow a new contributor.
