all the criticism has been said, all the takes been had.
-
RE: https://hails.org/@hailey/116657391001259044
all the criticism has been said, all the takes been had. the only metaphor i have been finding consistently useful for understanding what is happening with people and "AI" is addiction, and specifically gambling addiction.
@jonny holy crap this story gets worse by the day. Thank you very much for summing-up this aspect of the situation for a non-sw-engineering-person like me. 🫡
-
@jonny also why the hell would they write tests for a C program/library in Python? It makes no sense.
@jonny ... and why the everloving FUCK do these tests run as root
-
I think the modal situation here is that the people are reading none or very little of what is being generated by the LLM, so the tests have a special role: Tests function as the pull arm on the slot machine, you just generate until tests pass, and that's a jackpot. Obviously that's meaningless when the tests are meaningless, so tests take on a very different meaning and role in slot machine coding.
Previously we would write careful test conditions that were based off some real problem or an understanding of what the code under test did, and had a specific thing they were intended to protect against. Tests move slow and are designed to protect us against the things we know can go wrong. When we learn of a new wrong thing, we add a test.
LLM tests have the form of tests but don't do the same thing. They often test nothing, and are just expressions of truisms that the probabilistic text space explored while generating. They have strongly worded names but end up actually asserting that basic language features work as expected. Because it is not us writing tests for ourselves, where we only harm ourselves by making them weak, they function instead as a passively obfuscated justification for the code that the LLM generates. The user wants the tests to pass. The LLM provides.
The tests are theater: they are the play field for the slot machine. They are mild, surmountable, need to fail a few times to be plausible, but must eventually pass within the expected generation loop window to deliver the payout.
@jonny Referencing
1. @shauna post based on @DGI about power dynamics & dysfunction between imaginary labour(iML) & interpretive labour(iNL)-https://www.rethinkingpower.info/how-interpretive-labor-straddles-the-gap-between-rules-and-reality/
2. Power, chapter 4 of Mary Parker Follet’s Dynamic administration - https://mastodon.social/@dahukanna/110643444784446704Presuming Productivity(P)=(iML/iNL)
dysfunctional power-over tool imposition e.g. LLM, factory production,etc
- Imagined abstract: 1 LLM PR/0 review units= ∞P
- Interpreted reality: 1 LLM PR/>10 review units=0.1P
-https://mastodon.social/@dahukanna/113230734549577353 -
@jonny ... and why the everloving FUCK do these tests run as root
@fluffy@plush.city @jonny@neuromatch.social running tests as root is fucking wild
-
@jens, great article, thank you. Did you pull the lever "just one more time" and if so, did it get even worse?
@jonny, thank you for this thread and lots of your other threads on the topic.
Both help feeling that I'm not the ghost driver although these days there is lot of contraflow on my lane. Mostly at work where the AI fanboys/believers/addicts are at least way louder than the people trying to understand and keeping their code in maintainable shape.
-
@jens, great article, thank you. Did you pull the lever "just one more time" and if so, did it get even worse?
@jonny, thank you for this thread and lots of your other threads on the topic.
Both help feeling that I'm not the ghost driver although these days there is lot of contraflow on my lane. Mostly at work where the AI fanboys/believers/addicts are at least way louder than the people trying to understand and keeping their code in maintainable shape.
-
-
@jonny this whole thing is so bad that the only viable way seems to fork it before the LLM sloppening. It is a shame to see more and more foundational projects fall into the LLM trap.
And as always you hit the nail on the head with your deep dive and explanations. I love reading them.
I will use your observation on how for a LLM what is written is the same as what is happening.
@themipper @jonny
> It is a shame to see more and more foundational projects fall into the LLM trapThe one that breaks my heart is vim.
-
Here's an example from some code that was thrust at me this week. The rest of the tests try a bit harder to look like tests, but this one is perplexing.
What does it test? The function name suggests its a smoke test. LLMs love to call things smoke tests. That would suggest this would be an early-run test that fails loudly if some basic precondition - like having ffmpeg - fails. Or, I guess we are smoke testing the
ensure_ffmpegfunction? Anyway who knows. However we first check if ffmpeg or ffprobe are present, which is exactly whatensure_ffmpegdoes. If they aren't present, a warning tells us that ffmpeg/ffprobe are required for the video tests, which makes it seem like this should be a parameterizing test that controls which tests are run, which of course it does not do.So the test literally does nothing and cannot possibly fail, but says it does at least two things, because to an LLM something saying it does something is the same thing as it actually doing that thing.
@jonny (Un)charitable interpretation: it smoke tests whether the ensure_ffmpeg function is syntactically correct — which is a failure mode LLMs are actually concerned about.
-
@jonny also why the hell would they write tests for a C program/library in Python? It makes no sense.
@fluffy
Apparently all the tests for rsync are integration tests across bash rsync calls -
@KalenXI @jonny if you don't want to get lazy when using AI, is there really a use for it at all? I mean, it's been proven that reviewing code is much more difficult than writing it, so I'm finding it much more taxing to review slopcode than if I'd just write it myself.
Of course, that's adding the usual disclaimer that all this is not even relevant until the ethical and environmental shitshow of AI has been fixed.
@ainmosni @KalenXI @jonny With the caveat that the ethical problems with AI mean it's absolutely not worth the cost:
It looks like AIs are actually getting better at finding bugs and security holes.
So do your usual testing and code reviews, and then ask Claude to find any bugs you may have missed. It will give you some false positives but also some true ones.
Very different from having an LLM generate the code and a human try to fix it up.
Like Cory Doctorow's example: using an AI to give a second opinion on MRI scans (a centaur) makes scans more expensive but higher quality.
Having an AI analyse the scans at high speed and then getting some poor schmuck to try to spot its mistakes (reverse centaur) makes scans cheaper and lower quality, but at least there's a person with little power in the hierarchy who gets the blame for the problems.
Guess which one the people pouring trillions into AI want?
-
@jens @jonny At work "everyone has to use AI" according to C level people, despite all examples of things going wrong or at best "only" wasting lots of resources. Full hype mode.
LLMs are a tool. I just haven't figured out what for they are a reliable and reasonable choice.
For "How could problem X be addressed?", f.i. with a language I know very well, it might generate an answer with good points for me to look up and verify by myself in detail. Like a shortcut for a bunch of search queries.
-
So, look. One shot rewriting the whole test suite in another language is probably not great to do, but what happened here is so much worse than you are expecting.
https://github.com/RsyncProject/rsync/pull/903/
This does not "translate tests into pytest" or a unit testing framework, it writes its own testing framework where tests are whole python scripts that redefine basic test functions in every script. Surely there would be a single way to "run rsync and get the results" - nope, well, there is, but then every test file will randomly redefine its own
_run_and_capturefunction. So like now rsync needs a test suite for its test suite.If instead of telling an LLM to "rewrite the tests in python" you just searched "python testing" you would find the pytest docs. And then you would find examples. And then you could write fixtures to deduplicate all the prior shell script setup and teardown stuff, and so on. But since it was just "rewrite the tests in python" its now worse than before, and the odds of the rewrite actually being a 100% faithful translation are close to 0.
@jonny Ugh. I didn't realize at first which project this was. Then I looked at the repo. Yes, the road to hell IS paved with good intentions...
-
To a person, the whole purpose of the test is for it to fail when it should. That's an elemental part of writing good tests: they must fail before the patch, or else they provide no protection. We want protection from failure, that is good for us. We need tests to protect us because we can't possibly evaluate all the other parts of a complex system when we try to fix one part of it.
LLM slot machines change what tests mean - of course we still want the code to work good, but if we're not evaluating the code or the tests, then what the slot machine turns them into is just a high score and the jackpot condition. 130 new tests added, that means its good. They pass, that means I win.
The bugfix loop with LLMs defeats the purpose of automated tests and renders it no better than manual testing: you notice a bug, you yell at the LLM to fix it, you keep looking at the specific thing that's broken until its fixed, good robot, ship it. The changes don't have meaningful tests, and nothing else does either, so the slot machine loop repeats, bug->fix->win. Very velocity. Rocket fuel even.
@jonny It's a pattern I've been noticing all over.
Step 1: a process is created to measure something, like "does the software work right?" or "who do people want to be president?"
Step 2: There's an incentive for the people who perform and maintain the process to get a certain outcome, like good performance reviews or the guy you like being elected.
Step 3: Lacking the power to alter the thing being measured, the people in charge get creative with how they measure. -
@jonny Gambling doesn't attract me at all, never has, wonder if that's why I've been more resistant to it than others...
I'm kinda similar, the most I tried gambling was when I went to a gambling town for a wedding. I came out up but it just didn't do much for me.
Except poker, maybe because that involved people. That was fun.
-
@jonny It's a pattern I've been noticing all over.
Step 1: a process is created to measure something, like "does the software work right?" or "who do people want to be president?"
Step 2: There's an incentive for the people who perform and maintain the process to get a certain outcome, like good performance reviews or the guy you like being elected.
Step 3: Lacking the power to alter the thing being measured, the people in charge get creative with how they measure.@jonny It's those darn management cybernetics all over again. In order to make good software, we built a system of writing and passing tests—and (say it with me) the purpose of a system is what it does. The starting assumption that "did we make good software" would always be a critical test turned out to be false, because no system was constructed to keep it true.
-
@themipper @jonny
> It is a shame to see more and more foundational projects fall into the LLM trapThe one that breaks my heart is vim.
@JetSetIlly @jonny damn.
-
@jonny I have seen this pattern of which you speak when attempting to use LLMs to compare TCP Delayed-ACK implementations between BSD derived code bases. They generated output suggesting semantics that just weren't there, presumably based on how similarly named things were between each fork, but this was not obvious without reading the source for oneself in context. This went doubly for FreeBSD where there are multiple TCP functional blocks ("stacks").
@jonny Reminded of the "similarity quagmire" by changes Netflix guys are staging for FreeBSD TCP logs. My engineering notes from Feb, they did 80-90% of what I had in mind to do. The LLMs kept confabulating and conflating the meaning of "delayed_ack" with delacktime. "delayed_ack" is a variant ; its meaning is context dependent for macOS or between BSDs. it is neither a timer delta, nor is it an outstanding segment threshold counter; it can be all of those things. The LLMs did not infer this.
-
@jonny Reminded of the "similarity quagmire" by changes Netflix guys are staging for FreeBSD TCP logs. My engineering notes from Feb, they did 80-90% of what I had in mind to do. The LLMs kept confabulating and conflating the meaning of "delayed_ack" with delacktime. "delayed_ack" is a variant ; its meaning is context dependent for macOS or between BSDs. it is neither a timer delta, nor is it an outstanding segment threshold counter; it can be all of those things. The LLMs did not infer this.
@jonny The really bizarre thing is, that even if LLMs were capable of actual reasoning, they'd be considered to be committing category error in how they parse source code; logical fallacies then follow from that. Deep learning proponents also appear to have been doing this. The terminology describing LLMs and deep learning invites category error and cognitive dissonance. Geoffrey Hinton has a lot to answer for with his "It might be conscious" emergence-based woo.
-
I think the modal situation here is that the people are reading none or very little of what is being generated by the LLM, so the tests have a special role: Tests function as the pull arm on the slot machine, you just generate until tests pass, and that's a jackpot. Obviously that's meaningless when the tests are meaningless, so tests take on a very different meaning and role in slot machine coding.
Previously we would write careful test conditions that were based off some real problem or an understanding of what the code under test did, and had a specific thing they were intended to protect against. Tests move slow and are designed to protect us against the things we know can go wrong. When we learn of a new wrong thing, we add a test.
LLM tests have the form of tests but don't do the same thing. They often test nothing, and are just expressions of truisms that the probabilistic text space explored while generating. They have strongly worded names but end up actually asserting that basic language features work as expected. Because it is not us writing tests for ourselves, where we only harm ourselves by making them weak, they function instead as a passively obfuscated justification for the code that the LLM generates. The user wants the tests to pass. The LLM provides.
The tests are theater: they are the play field for the slot machine. They are mild, surmountable, need to fail a few times to be plausible, but must eventually pass within the expected generation loop window to deliver the payout.
@jonny nailed it
