all the criticism has been said, all the takes been had.
-
@poleguy
RIP on the shootout, hopefully the other team bought the beer and you got to pinch the other goalies cheek a bit. You'll get em next season@jonny indeed, that's the right feeling!
We have sponsorship from a brewery, so the locker room beer (and custom jerseys) are "free."
But we sat at the bar with the other team. It is just a game after all.
Both sides had a good time. And we had fans cheering for both sides. And kids crashing the locker room to celebrate despite the loss... We shared our NA options. Can't ask for more.
I'd love to engage more on this thread technically... I have thoughts. Maybe Monday.
-
I think the modal situation here is that the people are reading none or very little of what is being generated by the LLM, so the tests have a special role: Tests function as the pull arm on the slot machine, you just generate until tests pass, and that's a jackpot. Obviously that's meaningless when the tests are meaningless, so tests take on a very different meaning and role in slot machine coding.
Previously we would write careful test conditions that were based off some real problem or an understanding of what the code under test did, and had a specific thing they were intended to protect against. Tests move slow and are designed to protect us against the things we know can go wrong. When we learn of a new wrong thing, we add a test.
LLM tests have the form of tests but don't do the same thing. They often test nothing, and are just expressions of truisms that the probabilistic text space explored while generating. They have strongly worded names but end up actually asserting that basic language features work as expected. Because it is not us writing tests for ourselves, where we only harm ourselves by making them weak, they function instead as a passively obfuscated justification for the code that the LLM generates. The user wants the tests to pass. The LLM provides.
The tests are theater: they are the play field for the slot machine. They are mild, surmountable, need to fail a few times to be plausible, but must eventually pass within the expected generation loop window to deliver the payout.
-
@ainmosni
Well good, keep those walls up, they are protective. I am not so lucky and rely on discipline and observation of the impacts on others. It sets my alarm bells ringing to run for cover, but to understand why the things are happening around me the only means has been to feel it for myself, and I get it.@jonny @ainmosni Gambling (addiction) works on the so-called Variable Reinforcement Schedule.
The TL;DR of it is, results are random enough that even though it seems there may be a pattern, there isn't. You're pulled in because "one more time will show my pattern detection was right".
And since human brains are excellent pattern detection machines, every time this succeeds yields huge dopamine rewards.
I'm pissed off with the pattern, which is why I stop. But I can't deny its power.
-
So rsync rewriting all the tests puts the entire project in play. Now the entire protective surface has been sloshed through a layer of probability, so the loop must accelerate. Followup PRs add more carveouts with lengthy LLM justifications that sound perfectly plausible but amount to an erosion of the protective surface. We go from cumulative improvement to a random walk.
@jonny this whole thing is so bad that the only viable way seems to fork it before the LLM sloppening. It is a shame to see more and more foundational projects fall into the LLM trap.
And as always you hit the nail on the head with your deep dive and explanations. I love reading them.
I will use your observation on how for a LLM what is written is the same as what is happening.
-
Here's an example from some code that was thrust at me this week. The rest of the tests try a bit harder to look like tests, but this one is perplexing.
What does it test? The function name suggests its a smoke test. LLMs love to call things smoke tests. That would suggest this would be an early-run test that fails loudly if some basic precondition - like having ffmpeg - fails. Or, I guess we are smoke testing the
ensure_ffmpegfunction? Anyway who knows. However we first check if ffmpeg or ffprobe are present, which is exactly whatensure_ffmpegdoes. If they aren't present, a warning tells us that ffmpeg/ffprobe are required for the video tests, which makes it seem like this should be a parameterizing test that controls which tests are run, which of course it does not do.So the test literally does nothing and cannot possibly fail, but says it does at least two things, because to an LLM something saying it does something is the same thing as it actually doing that thing.
@jonny Of course it's a smoke test - as in "smoke and mirrors"
WTAF.
-
So rsync rewriting all the tests puts the entire project in play. Now the entire protective surface has been sloshed through a layer of probability, so the loop must accelerate. Followup PRs add more carveouts with lengthy LLM justifications that sound perfectly plausible but amount to an erosion of the protective surface. We go from cumulative improvement to a random walk.
@jonny also why the hell would they write tests for a C program/library in Python? It makes no sense.
-
RE: https://hails.org/@hailey/116657391001259044
all the criticism has been said, all the takes been had. the only metaphor i have been finding consistently useful for understanding what is happening with people and "AI" is addiction, and specifically gambling addiction.
@jonny holy crap this story gets worse by the day. Thank you very much for summing-up this aspect of the situation for a non-sw-engineering-person like me. 🫡
-
@jonny also why the hell would they write tests for a C program/library in Python? It makes no sense.
@jonny ... and why the everloving FUCK do these tests run as root
-
I think the modal situation here is that the people are reading none or very little of what is being generated by the LLM, so the tests have a special role: Tests function as the pull arm on the slot machine, you just generate until tests pass, and that's a jackpot. Obviously that's meaningless when the tests are meaningless, so tests take on a very different meaning and role in slot machine coding.
Previously we would write careful test conditions that were based off some real problem or an understanding of what the code under test did, and had a specific thing they were intended to protect against. Tests move slow and are designed to protect us against the things we know can go wrong. When we learn of a new wrong thing, we add a test.
LLM tests have the form of tests but don't do the same thing. They often test nothing, and are just expressions of truisms that the probabilistic text space explored while generating. They have strongly worded names but end up actually asserting that basic language features work as expected. Because it is not us writing tests for ourselves, where we only harm ourselves by making them weak, they function instead as a passively obfuscated justification for the code that the LLM generates. The user wants the tests to pass. The LLM provides.
The tests are theater: they are the play field for the slot machine. They are mild, surmountable, need to fail a few times to be plausible, but must eventually pass within the expected generation loop window to deliver the payout.
@jonny Referencing
1. @shauna post based on @DGI about power dynamics & dysfunction between imaginary labour(iML) & interpretive labour(iNL)-https://www.rethinkingpower.info/how-interpretive-labor-straddles-the-gap-between-rules-and-reality/
2. Power, chapter 4 of Mary Parker Follet’s Dynamic administration - https://mastodon.social/@dahukanna/110643444784446704Presuming Productivity(P)=(iML/iNL)
dysfunctional power-over tool imposition e.g. LLM, factory production,etc
- Imagined abstract: 1 LLM PR/0 review units= ∞P
- Interpreted reality: 1 LLM PR/>10 review units=0.1P
-https://mastodon.social/@dahukanna/113230734549577353 -
@jonny ... and why the everloving FUCK do these tests run as root
@fluffy@plush.city @jonny@neuromatch.social running tests as root is fucking wild
-
@jens, great article, thank you. Did you pull the lever "just one more time" and if so, did it get even worse?
@jonny, thank you for this thread and lots of your other threads on the topic.
Both help feeling that I'm not the ghost driver although these days there is lot of contraflow on my lane. Mostly at work where the AI fanboys/believers/addicts are at least way louder than the people trying to understand and keeping their code in maintainable shape.
-
@jens, great article, thank you. Did you pull the lever "just one more time" and if so, did it get even worse?
@jonny, thank you for this thread and lots of your other threads on the topic.
Both help feeling that I'm not the ghost driver although these days there is lot of contraflow on my lane. Mostly at work where the AI fanboys/believers/addicts are at least way louder than the people trying to understand and keeping their code in maintainable shape.
-
-
@jonny this whole thing is so bad that the only viable way seems to fork it before the LLM sloppening. It is a shame to see more and more foundational projects fall into the LLM trap.
And as always you hit the nail on the head with your deep dive and explanations. I love reading them.
I will use your observation on how for a LLM what is written is the same as what is happening.
@themipper @jonny
> It is a shame to see more and more foundational projects fall into the LLM trapThe one that breaks my heart is vim.
-
Here's an example from some code that was thrust at me this week. The rest of the tests try a bit harder to look like tests, but this one is perplexing.
What does it test? The function name suggests its a smoke test. LLMs love to call things smoke tests. That would suggest this would be an early-run test that fails loudly if some basic precondition - like having ffmpeg - fails. Or, I guess we are smoke testing the
ensure_ffmpegfunction? Anyway who knows. However we first check if ffmpeg or ffprobe are present, which is exactly whatensure_ffmpegdoes. If they aren't present, a warning tells us that ffmpeg/ffprobe are required for the video tests, which makes it seem like this should be a parameterizing test that controls which tests are run, which of course it does not do.So the test literally does nothing and cannot possibly fail, but says it does at least two things, because to an LLM something saying it does something is the same thing as it actually doing that thing.
@jonny (Un)charitable interpretation: it smoke tests whether the ensure_ffmpeg function is syntactically correct — which is a failure mode LLMs are actually concerned about.
-
@jonny also why the hell would they write tests for a C program/library in Python? It makes no sense.
@fluffy
Apparently all the tests for rsync are integration tests across bash rsync calls -
@KalenXI @jonny if you don't want to get lazy when using AI, is there really a use for it at all? I mean, it's been proven that reviewing code is much more difficult than writing it, so I'm finding it much more taxing to review slopcode than if I'd just write it myself.
Of course, that's adding the usual disclaimer that all this is not even relevant until the ethical and environmental shitshow of AI has been fixed.
@ainmosni @KalenXI @jonny With the caveat that the ethical problems with AI mean it's absolutely not worth the cost:
It looks like AIs are actually getting better at finding bugs and security holes.
So do your usual testing and code reviews, and then ask Claude to find any bugs you may have missed. It will give you some false positives but also some true ones.
Very different from having an LLM generate the code and a human try to fix it up.
Like Cory Doctorow's example: using an AI to give a second opinion on MRI scans (a centaur) makes scans more expensive but higher quality.
Having an AI analyse the scans at high speed and then getting some poor schmuck to try to spot its mistakes (reverse centaur) makes scans cheaper and lower quality, but at least there's a person with little power in the hierarchy who gets the blame for the problems.
Guess which one the people pouring trillions into AI want?
-
@jens @jonny At work "everyone has to use AI" according to C level people, despite all examples of things going wrong or at best "only" wasting lots of resources. Full hype mode.
LLMs are a tool. I just haven't figured out what for they are a reliable and reasonable choice.
For "How could problem X be addressed?", f.i. with a language I know very well, it might generate an answer with good points for me to look up and verify by myself in detail. Like a shortcut for a bunch of search queries.
-
So, look. One shot rewriting the whole test suite in another language is probably not great to do, but what happened here is so much worse than you are expecting.
https://github.com/RsyncProject/rsync/pull/903/
This does not "translate tests into pytest" or a unit testing framework, it writes its own testing framework where tests are whole python scripts that redefine basic test functions in every script. Surely there would be a single way to "run rsync and get the results" - nope, well, there is, but then every test file will randomly redefine its own
_run_and_capturefunction. So like now rsync needs a test suite for its test suite.If instead of telling an LLM to "rewrite the tests in python" you just searched "python testing" you would find the pytest docs. And then you would find examples. And then you could write fixtures to deduplicate all the prior shell script setup and teardown stuff, and so on. But since it was just "rewrite the tests in python" its now worse than before, and the odds of the rewrite actually being a 100% faithful translation are close to 0.
@jonny Ugh. I didn't realize at first which project this was. Then I looked at the repo. Yes, the road to hell IS paved with good intentions...
-
To a person, the whole purpose of the test is for it to fail when it should. That's an elemental part of writing good tests: they must fail before the patch, or else they provide no protection. We want protection from failure, that is good for us. We need tests to protect us because we can't possibly evaluate all the other parts of a complex system when we try to fix one part of it.
LLM slot machines change what tests mean - of course we still want the code to work good, but if we're not evaluating the code or the tests, then what the slot machine turns them into is just a high score and the jackpot condition. 130 new tests added, that means its good. They pass, that means I win.
The bugfix loop with LLMs defeats the purpose of automated tests and renders it no better than manual testing: you notice a bug, you yell at the LLM to fix it, you keep looking at the specific thing that's broken until its fixed, good robot, ship it. The changes don't have meaningful tests, and nothing else does either, so the slot machine loop repeats, bug->fix->win. Very velocity. Rocket fuel even.
@jonny It's a pattern I've been noticing all over.
Step 1: a process is created to measure something, like "does the software work right?" or "who do people want to be president?"
Step 2: There's an incentive for the people who perform and maintain the process to get a certain outcome, like good performance reviews or the guy you like being elected.
Step 3: Lacking the power to alter the thing being measured, the people in charge get creative with how they measure.
