Yes, no (built-in) AI is now a feature
-
I have, on several occasions.
One that stands out is when I asked a bot (I think Claude Haiku) a question about a programming topic I'm not entirely familiar with. It gave me 10 answers, 4 of which were obviously wrong based on the self-contained test case I presented it with, and 4 more of which immediately proved wrong upon cursory testing.
It was obvious that the bot was just making s**t up and had no idea what it was talking about.
@argv_minus_one @grvsmth @wobweger @libreoffice @moskitokoenig
> One that stands out is when I asked a bot (I think Claude Haiku) a question about a programming topic I'm not entirely familiar with. It gave me 10 answers, 4 of which were obviously wrong based on the self-contained test case I presented it
I am curious.
May I have the question? I can try it out here
-
Your being astounded does not mean LLMs have surpassed humans, only that they've surpassed you.
Anyway, Greg K-H is talking about using LLMs to find bugs, like a static analyzer. That's grand and all, but you're talking about vibe code, and that's not what he's talking about.
I'm also not impressed with the high false-positive rate and psychologically-manipulative behavior Greg describes having experienced in his interactions with bots. Maybe he finds that tolerable, but I don't.
-
@Garonenur @argv_minus_one @wobweger @libreoffice @grvsmth @moskitokoenig
Your comment about their output being shit is two years out of date
These days the code they produce is often very good.
Times are changing, and the development since 2022 has been astounding
The last nine months alone has seen huge advances.
@worik @argv_minus_one @wobweger @libreoffice @grvsmth @moskitokoenig I have to read and review the code right today.
And it sucks.
It reuses old fashioned paradigms, because the old shit is more prevalent in the training data.
AND the users of those LLMs all start to become lazy and just believe what hallucinations the machine dishes them.
Code quality was terrible before, it will only go down now. -
@argv_minus_one @grvsmth @wobweger @libreoffice @moskitokoenig
That is the exact opposite of my experience
You sure that you are not making it up?
Well, I just looked over the lengthy transcripts of the chats I had with two different bots on the topic (Mistral and GPT-5.4-mini, as it turns out), and I *was* going to go over them and redact them…
…but then it occurred to me that you're probably just going to have your bot “summarize” the conversations anyway…
…in which case this is a waste of my time, so I'm not going to bother.
-
Well, I just looked over the lengthy transcripts of the chats I had with two different bots on the topic (Mistral and GPT-5.4-mini, as it turns out), and I *was* going to go over them and redact them…
…but then it occurred to me that you're probably just going to have your bot “summarize” the conversations anyway…
…in which case this is a waste of my time, so I'm not going to bother.
I have, however, gone ahead and saved the transcripts, for my own future amusement and/or to bring receipts to someone who's actually willing to read them. So thanks for reminding me to do that.
-
I have, however, gone ahead and saved the transcripts, for my own future amusement and/or to bring receipts to someone who's actually willing to read them. So thanks for reminding me to do that.
@argv_minus_one @grvsmth @wobweger @moskitokoenig
I am glad you did not do the work correcting the transcripts, that is not what interests me.
What interests me is the question. From memory (interface here will not let me rep.y and look at thread simultaneously easilly) the question was a computing problem, you got ten answers and four were correct.
I am interested. I
-
Your being astounded does not mean LLMs have surpassed humans, only that they've surpassed you.
Anyway, Greg K-H is talking about using LLMs to find bugs, like a static analyzer. That's grand and all, but you're talking about vibe code, and that's not what he's talking about.
@argv_minus_one @grvsmth @Garonenur @wobweger @moskitokoenig
> Greg K-H is talking about using LLMs to find bugs,
More than that, he urges people to develop patches, using LLM prompts, so the bug report comes with a solution (unless I my recall is fault.ty, I watched said video yesterday but only once)
His point is that doing so exposes that the LLM generates false positives, and propper, simple, use of LLMs can reduce them
-
@argv_minus_one @grvsmth @Garonenur @wobweger @moskitokoenig
> Greg K-H is talking about using LLMs to find bugs,
More than that, he urges people to develop patches, using LLM prompts, so the bug report comes with a solution (unless I my recall is fault.ty, I watched said video yesterday but only once)
His point is that doing so exposes that the LLM generates false positives, and propper, simple, use of LLMs can reduce them
@argv_minus_one @grvsmth @Garonenur @wobweger @moskitokoenig
I think the point, Greg's in the video, and mine here, is these are tools and need to be wielded. Careless use has bad results
Careful use increases the reach of the user
-
@argv_minus_one @grvsmth @wobweger @moskitokoenig
I am glad you did not do the work correcting the transcripts, that is not what interests me.
What interests me is the question. From memory (interface here will not let me rep.y and look at thread simultaneously easilly) the question was a computing problem, you got ten answers and four were correct.
I am interested. I
You're nothing if not persistent. Very well.
I looked over the logs. Looks like nothing to redact. Here they are: https://files.catbox.moe/wzzpci.txt
Mistral screwed up 8 times and was incidentally right about 1 thing but that wasn't the actual problem.
GPT screwed up 12 times, said 1 thing that's maybe right but I'm not even sure, and made 1 “optional” suggestion that turned out to be the real solution to the problem.
-
You're nothing if not persistent. Very well.
I looked over the logs. Looks like nothing to redact. Here they are: https://files.catbox.moe/wzzpci.txt
Mistral screwed up 8 times and was incidentally right about 1 thing but that wasn't the actual problem.
GPT screwed up 12 times, said 1 thing that's maybe right but I'm not even sure, and made 1 “optional” suggestion that turned out to be the real solution to the problem.
To be clear, GPT's suggestion to drain input from the server side of the socket was the actual solution. My code worked completely after that.
The suggestion to avoid partial writes was a sound one. I was under the mistaken impression that Java does its own retrying and partial writes therefore can't happen. But this was not the real problem; it was only incidental.
-
To be clear, GPT's suggestion to drain input from the server side of the socket was the actual solution. My code worked completely after that.
The suggestion to avoid partial writes was a sound one. I was under the mistaken impression that Java does its own retrying and partial writes therefore can't happen. But this was not the real problem; it was only incidental.
Although GPT finally did give me the actual solution, I'm still calling this an epic fail, in light of the vast volume of utter crap that both bots threw at me.
As I said before, it's clear that they were both just making s**t up, had no clue, and accidentally stumbled upon the correct answer.
-
Although GPT finally did give me the actual solution, I'm still calling this an epic fail, in light of the vast volume of utter crap that both bots threw at me.
As I said before, it's clear that they were both just making s**t up, had no clue, and accidentally stumbled upon the correct answer.
I would like to add that both bots tried to convince me to vastly increase the complexity of this tiny HTTP server by adding Content-Length, response chunking, and more thorough request parsing.
The whole point of this HTTP server, as should have been obvious from the minimalism of its code, was to serve one file at one URL-path with as small an attack surface as possible.
AI loves unnecessary complexity, and that is very dangerous security-wise.
-
Although GPT finally did give me the actual solution, I'm still calling this an epic fail, in light of the vast volume of utter crap that both bots threw at me.
As I said before, it's clear that they were both just making s**t up, had no clue, and accidentally stumbled upon the correct answer.
@argv_minus_one @grvsmth @wobweger @moskitokoenig
Using duck.ai is great for simple requests. I find >200 line prompts challenge it quite a bit.
I am unfamiliar with Java (been many decades since I cared about Java - I have no use for it currently) but I am going to see what my agents do with it
-
I would like to add that both bots tried to convince me to vastly increase the complexity of this tiny HTTP server by adding Content-Length, response chunking, and more thorough request parsing.
The whole point of this HTTP server, as should have been obvious from the minimalism of its code, was to serve one file at one URL-path with as small an attack surface as possible.
AI loves unnecessary complexity, and that is very dangerous security-wise.
@argv_minus_one @grvsmth @wobweger @moskitokoenig
> both bots tried to convince me to vastly increase the complexity of this tiny HTTP server
Yes, I am familiar with that! DRY is sometimes a foreign concept to agents - they need reminding.
This is why "vibecoding" is no replacement for programmers
On a tangent Bruce Schneier has an interesting take on this (Bruce was a heavy sceptic of "AI" in 2022, I think it would be fair to say he has adjusted his views).
-
@argv_minus_one @grvsmth @Garonenur @wobweger @moskitokoenig
I think the point, Greg's in the video, and mine here, is these are tools and need to be wielded. Careless use has bad results
Careful use increases the reach of the user
The problem with this is that AI itself actively encourages careless use, in two ways:
1. It creates the illusion of being easy to use. Ask it to write some code for you, and it does so. Only if you're both skilled and vigilant will you notice everything that's wrong with the code.
[continued]
-
The problem with this is that AI itself actively encourages careless use, in two ways:
1. It creates the illusion of being easy to use. Ask it to write some code for you, and it does so. Only if you're both skilled and vigilant will you notice everything that's wrong with the code.
[continued]
2. It kisses your ass. It appeals to your ego. It encourages you to use it more.
That behavior is downright manipulative, and reminds me of con artists and drug dealers.
I avoid AI not only because I find it largely useless, but also because I consider its manipulative behavior extremely dangerous. If I'm exposed to it long enough, I don't know if I can stop it from successfully manipulating me. Probably not.
-
The problem with this is that AI itself actively encourages careless use, in two ways:
1. It creates the illusion of being easy to use. Ask it to write some code for you, and it does so. Only if you're both skilled and vigilant will you notice everything that's wrong with the code.
[continued]
@argv_minus_one @grvsmth @Garonenur @wobweger @moskitokoenig
> 1. It creates the illusion of being easy to use
This is a **huge** problem. It does lower one of the barriers of entry into the software producing world - it can cut a lot of code quickly, it will compile, it will run sort of like what you want.
All of the other barriers, including but not limited too: Understanding the problem domain, logical thinking, organised thinking... It does not help with those in such an easy way.
Yes
-
@argv_minus_one @grvsmth @Garonenur @wobweger @moskitokoenig
> 1. It creates the illusion of being easy to use
This is a **huge** problem. It does lower one of the barriers of entry into the software producing world - it can cut a lot of code quickly, it will compile, it will run sort of like what you want.
All of the other barriers, including but not limited too: Understanding the problem domain, logical thinking, organised thinking... It does not help with those in such an easy way.
Yes
Yes, and that's the hard part of programming anyway.
I don't need help writing code. I have perfectly serviceable hands and keyboard with which to do so.
It would be helpful if AI could reason about code and invent a novel solution to the problem at hand…but if it can do that, then it also can, and probably will, go full Skynet and kill us all.
-
2. It kisses your ass. It appeals to your ego. It encourages you to use it more.
That behavior is downright manipulative, and reminds me of con artists and drug dealers.
I avoid AI not only because I find it largely useless, but also because I consider its manipulative behavior extremely dangerous. If I'm exposed to it long enough, I don't know if I can stop it from successfully manipulating me. Probably not.
@argv_minus_one @grvsmth @Garonenur @wobweger @moskitokoenig
> 2. It kisses your ass. It appeals to your ego. It encourages you to use it more.
I feel so old! Yes, I am irritated by the subsequence of the responses. But I think it is going out of fashion.
The "encourages you to use it more". Anything that produces results is like that, true? I do not get the sense that I am being manipulated, like I see from, say, the effect of Youtube on my four and six year old grandchildren (shiver)
-
@argv_minus_one @grvsmth @wobweger @moskitokoenig
> both bots tried to convince me to vastly increase the complexity of this tiny HTTP server
Yes, I am familiar with that! DRY is sometimes a foreign concept to agents - they need reminding.
This is why "vibecoding" is no replacement for programmers
On a tangent Bruce Schneier has an interesting take on this (Bruce was a heavy sceptic of "AI" in 2022, I think it would be fair to say he has adjusted his views).
This is the KISS principle, not DRY. But yes, bots seem pretty bad at DRY too.
And to be fair to bots, a lot of human programmers are also quite bad at both of these principles. “Minimize attack surface” may as well be a foreign language to most people.
Which I guess explains why bots behave this way. Because that's who they learned from.
