Project Glasswing:
-
Project Glasswing:
Claiming to have found 26 thousand real vulnerabilities but only 0.8% of them have resulted in a real fix in a real project after five months is dire. They blame it on the human independent review bottleneck, but human experts being paid for their time definitely have a higher throughput than that when working with data that’s actually actionable.
The assigned-at-Claude severity ratings are also dire. It assigns “high” or “critical” to 91% of findings. Most findings in the real world are low or medium. This should be especially true when using a magic machine to shake out every last little issue that was overlooked by humans focused on the biggest risks.
[Edit: I should be more careful and note that this figure is calculated only from findings which have received a second opinion from project maintainers, which is much higher than the 0.8% fixed rate but much less than the entire dataset, so there’s probably bias towards reviewing those with critical ratings first. However, the maintainers found the high/critical rate to be quite inflated.]
Together this implies it’s generating thousands of trivial or nonsensical findings and labeling them HIGH DANGER CRITICAL MUST FIX, and the human independent verifiers are sifting for the rare needle in this haystack worth passing on. This isn’t really an improvement over the high-noise automated scanners we already had
(This is a corporate blog of someone with their own vulnerability management services to sell, so apply an appropriate number of grains of salt to their analysis. Filter keywords: AI LLM Anthropic)
@0xabad1dea This is my read as well but it is very good to have your judgement on this as well.
I don't think Twisted is in Glasswing's purview but this lines *right* up with the reports we have gotten. Overwhelming, but fewer than advertised, considerably less serious than the bots rate them, and, shall we say, poor quality remediations that would be unusable even if our policy were to accept slop fixes
-
@0xabad1dea This is my read as well but it is very good to have your judgement on this as well.
I don't think Twisted is in Glasswing's purview but this lines *right* up with the reports we have gotten. Overwhelming, but fewer than advertised, considerably less serious than the bots rate them, and, shall we say, poor quality remediations that would be unusable even if our policy were to accept slop fixes
@0xabad1dea Even more than my own "adversarial communication" hypothesis what this is starting to remind me of is the proposed concept of chaffing and winnowing <https://en.wikipedia.org/wiki/Chaffing_and_winnowing>. Bots emit the chaff, humans winnow to the actual useful information, nearly all of the useful effort is in the winnowing but the bots get the credit for having "generated" the solution.
-
@0xabad1dea Even more than my own "adversarial communication" hypothesis what this is starting to remind me of is the proposed concept of chaffing and winnowing <https://en.wikipedia.org/wiki/Chaffing_and_winnowing>. Bots emit the chaff, humans winnow to the actual useful information, nearly all of the useful effort is in the winnowing but the bots get the credit for having "generated" the solution.
@0xabad1dea It's like inverse responsibility-laundering. Instead of having humans take the blame for its mistakes, we're supposed to let the bots take the credit for all the humans' work.
-
@0xabad1dea It's like inverse responsibility-laundering. Instead of having humans take the blame for its mistakes, we're supposed to let the bots take the credit for all the humans' work.
@0xabad1dea This sucks.
-
Project Glasswing:
Claiming to have found 26 thousand real vulnerabilities but only 0.8% of them have resulted in a real fix in a real project after five months is dire. They blame it on the human independent review bottleneck, but human experts being paid for their time definitely have a higher throughput than that when working with data that’s actually actionable.
The assigned-at-Claude severity ratings are also dire. It assigns “high” or “critical” to 91% of findings. Most findings in the real world are low or medium. This should be especially true when using a magic machine to shake out every last little issue that was overlooked by humans focused on the biggest risks.
[Edit: I should be more careful and note that this figure is calculated only from findings which have received a second opinion from project maintainers, which is much higher than the 0.8% fixed rate but much less than the entire dataset, so there’s probably bias towards reviewing those with critical ratings first. However, the maintainers found the high/critical rate to be quite inflated.]
Together this implies it’s generating thousands of trivial or nonsensical findings and labeling them HIGH DANGER CRITICAL MUST FIX, and the human independent verifiers are sifting for the rare needle in this haystack worth passing on. This isn’t really an improvement over the high-noise automated scanners we already had
(This is a corporate blog of someone with their own vulnerability management services to sell, so apply an appropriate number of grains of salt to their analysis. Filter keywords: AI LLM Anthropic)
@0xabad1dea This is interesting because the result of this seems to be some pretty excellent fear-mongering that I suspect is the biggest reason so many things (like the Linux kernel) have decided to integrate such tools into processes that really shouldn't have them directly integrated. These things fire off so many "critical warning! Dangerous! Be scared!" warnings that people kneejerk into panic.
-
Project Glasswing:
Claiming to have found 26 thousand real vulnerabilities but only 0.8% of them have resulted in a real fix in a real project after five months is dire. They blame it on the human independent review bottleneck, but human experts being paid for their time definitely have a higher throughput than that when working with data that’s actually actionable.
The assigned-at-Claude severity ratings are also dire. It assigns “high” or “critical” to 91% of findings. Most findings in the real world are low or medium. This should be especially true when using a magic machine to shake out every last little issue that was overlooked by humans focused on the biggest risks.
[Edit: I should be more careful and note that this figure is calculated only from findings which have received a second opinion from project maintainers, which is much higher than the 0.8% fixed rate but much less than the entire dataset, so there’s probably bias towards reviewing those with critical ratings first. However, the maintainers found the high/critical rate to be quite inflated.]
Together this implies it’s generating thousands of trivial or nonsensical findings and labeling them HIGH DANGER CRITICAL MUST FIX, and the human independent verifiers are sifting for the rare needle in this haystack worth passing on. This isn’t really an improvement over the high-noise automated scanners we already had
(This is a corporate blog of someone with their own vulnerability management services to sell, so apply an appropriate number of grains of salt to their analysis. Filter keywords: AI LLM Anthropic)
@0xabad1dea Just before they did the ‘security’ stuff, someone sent me a ‘comprehensive code review’ of one of my projects. Most of the performance things were lifted directly from open issues I’d written, but one that it flagged as high importance was an intentional design decision.
It pointed out that one operation does a broadcast cache invalidation that invalidates all caches, rather than the subset that it actually needs to. The recommendation was to switch to having fine-grained cache state tracking. The problem is, that actually made cache accesses more expensive on the most performance-critical path in the system. Cache invalidations happened less than once every million accesses in typical programs, so this was an incredibly unusual case and optimising for it hurt performance everywhere else.
But, because it didn’t actually understand the code or the context, it saw a pattern that was slow and matched it to a high-severity warning.
I would expect it to do exactly the same thing with security reports. Memory-safety bug in a code path that can be reached only via trusted data? Highest severity!
-
@0xabad1dea It's like inverse responsibility-laundering. Instead of having humans take the blame for its mistakes, we're supposed to let the bots take the credit for all the humans' work.
@glyph @0xabad1dea which aligns with Cory Doctorow’s application of the idea of centaurs & “reverse centaurs” to the combination of humans & spicy autocarrots – the low-value high-volume firehose of slopped vuln discoveries turns humans into reverse centaurs, which is the opposite of what we expect & want our technologies to do
-
Project Glasswing:
Claiming to have found 26 thousand real vulnerabilities but only 0.8% of them have resulted in a real fix in a real project after five months is dire. They blame it on the human independent review bottleneck, but human experts being paid for their time definitely have a higher throughput than that when working with data that’s actually actionable.
The assigned-at-Claude severity ratings are also dire. It assigns “high” or “critical” to 91% of findings. Most findings in the real world are low or medium. This should be especially true when using a magic machine to shake out every last little issue that was overlooked by humans focused on the biggest risks.
[Edit: I should be more careful and note that this figure is calculated only from findings which have received a second opinion from project maintainers, which is much higher than the 0.8% fixed rate but much less than the entire dataset, so there’s probably bias towards reviewing those with critical ratings first. However, the maintainers found the high/critical rate to be quite inflated.]
Together this implies it’s generating thousands of trivial or nonsensical findings and labeling them HIGH DANGER CRITICAL MUST FIX, and the human independent verifiers are sifting for the rare needle in this haystack worth passing on. This isn’t really an improvement over the high-noise automated scanners we already had
(This is a corporate blog of someone with their own vulnerability management services to sell, so apply an appropriate number of grains of salt to their analysis. Filter keywords: AI LLM Anthropic)
@0xabad1dea in my experience of getting bug reports from Claude, it loves to classify things as use-after-free, I've yet to find a correctly identified security vulnerability
-
Project Glasswing:
Claiming to have found 26 thousand real vulnerabilities but only 0.8% of them have resulted in a real fix in a real project after five months is dire. They blame it on the human independent review bottleneck, but human experts being paid for their time definitely have a higher throughput than that when working with data that’s actually actionable.
The assigned-at-Claude severity ratings are also dire. It assigns “high” or “critical” to 91% of findings. Most findings in the real world are low or medium. This should be especially true when using a magic machine to shake out every last little issue that was overlooked by humans focused on the biggest risks.
[Edit: I should be more careful and note that this figure is calculated only from findings which have received a second opinion from project maintainers, which is much higher than the 0.8% fixed rate but much less than the entire dataset, so there’s probably bias towards reviewing those with critical ratings first. However, the maintainers found the high/critical rate to be quite inflated.]
Together this implies it’s generating thousands of trivial or nonsensical findings and labeling them HIGH DANGER CRITICAL MUST FIX, and the human independent verifiers are sifting for the rare needle in this haystack worth passing on. This isn’t really an improvement over the high-noise automated scanners we already had
(This is a corporate blog of someone with their own vulnerability management services to sell, so apply an appropriate number of grains of salt to their analysis. Filter keywords: AI LLM Anthropic)
There's a Norwegian expression I'm reminded of. "'Mykje skrik og lite ull,' sa mannen, han klipte grisen." Or, in English, "'A lot of noise and not much wool,' said the man who sheared his pig."
-
@0xabad1dea This is my read as well but it is very good to have your judgement on this as well.
I don't think Twisted is in Glasswing's purview but this lines *right* up with the reports we have gotten. Overwhelming, but fewer than advertised, considerably less serious than the bots rate them, and, shall we say, poor quality remediations that would be unusable even if our policy were to accept slop fixes
I was the unfortunate witness of a Fable-generated review of a game engine prototype and a non- negligible portions of output tokens were spent on how the offline video game might *gasp* not be memory-safe.
But it congratulated the author for their "serious warning posture" of compiling with -Wall -Werror so there's that.
I think you could probably point it at Hamlet and it would identify at least a couple vulns. -
Project Glasswing:
Claiming to have found 26 thousand real vulnerabilities but only 0.8% of them have resulted in a real fix in a real project after five months is dire. They blame it on the human independent review bottleneck, but human experts being paid for their time definitely have a higher throughput than that when working with data that’s actually actionable.
The assigned-at-Claude severity ratings are also dire. It assigns “high” or “critical” to 91% of findings. Most findings in the real world are low or medium. This should be especially true when using a magic machine to shake out every last little issue that was overlooked by humans focused on the biggest risks.
[Edit: I should be more careful and note that this figure is calculated only from findings which have received a second opinion from project maintainers, which is much higher than the 0.8% fixed rate but much less than the entire dataset, so there’s probably bias towards reviewing those with critical ratings first. However, the maintainers found the high/critical rate to be quite inflated.]
Together this implies it’s generating thousands of trivial or nonsensical findings and labeling them HIGH DANGER CRITICAL MUST FIX, and the human independent verifiers are sifting for the rare needle in this haystack worth passing on. This isn’t really an improvement over the high-noise automated scanners we already had
(This is a corporate blog of someone with their own vulnerability management services to sell, so apply an appropriate number of grains of salt to their analysis. Filter keywords: AI LLM Anthropic)
Anthropic needs another pallet of cash to keep the gas turbines spinning, so, WOLF BIG BADY BAD BAD WOLF!!!1!1Q
It's a VC cash plea posing as an actionable threat assessment. Anthropic's core talent is turning capital into waste heat.
-
@0xabad1dea Just before they did the ‘security’ stuff, someone sent me a ‘comprehensive code review’ of one of my projects. Most of the performance things were lifted directly from open issues I’d written, but one that it flagged as high importance was an intentional design decision.
It pointed out that one operation does a broadcast cache invalidation that invalidates all caches, rather than the subset that it actually needs to. The recommendation was to switch to having fine-grained cache state tracking. The problem is, that actually made cache accesses more expensive on the most performance-critical path in the system. Cache invalidations happened less than once every million accesses in typical programs, so this was an incredibly unusual case and optimising for it hurt performance everywhere else.
But, because it didn’t actually understand the code or the context, it saw a pattern that was slow and matched it to a high-severity warning.
I would expect it to do exactly the same thing with security reports. Memory-safety bug in a code path that can be reached only via trusted data? Highest severity!
@david_chisnall @0xabad1dea Revocation of capability or RBAC access control *behind* the capability grant does present an exception to usual received wisdom for soft state vs hard state, recalling Noel Chiappa's original note on the subject; it's closer to his definition of "critical state".
An LLM is not expected to understand this, paraphrasing the original UNIX source code comment.
-
Project Glasswing:
Claiming to have found 26 thousand real vulnerabilities but only 0.8% of them have resulted in a real fix in a real project after five months is dire. They blame it on the human independent review bottleneck, but human experts being paid for their time definitely have a higher throughput than that when working with data that’s actually actionable.
The assigned-at-Claude severity ratings are also dire. It assigns “high” or “critical” to 91% of findings. Most findings in the real world are low or medium. This should be especially true when using a magic machine to shake out every last little issue that was overlooked by humans focused on the biggest risks.
[Edit: I should be more careful and note that this figure is calculated only from findings which have received a second opinion from project maintainers, which is much higher than the 0.8% fixed rate but much less than the entire dataset, so there’s probably bias towards reviewing those with critical ratings first. However, the maintainers found the high/critical rate to be quite inflated.]
Together this implies it’s generating thousands of trivial or nonsensical findings and labeling them HIGH DANGER CRITICAL MUST FIX, and the human independent verifiers are sifting for the rare needle in this haystack worth passing on. This isn’t really an improvement over the high-noise automated scanners we already had
(This is a corporate blog of someone with their own vulnerability management services to sell, so apply an appropriate number of grains of salt to their analysis. Filter keywords: AI LLM Anthropic)
@0xabad1dea Internally at work we had a shit-ton of Glasswing findings. I got 5 sev-2 tickets for Glasswing issues. All on a project we had shut down 18 months prior. Of the 5, 4 were junk. They were in code that was only a partial implementation of a new feature, and we had stopped the entire project before the feature was finished.
The fifth was really subtle and interesting. It was legit, and something I had wondered whether it was possible. I checked it out and it was definitely true and exploitable the way it described. Shame the project had been shut down for 18 months.
I ended up accepting its PRs and then marking all the code repos as deprecated just to shut the bot up. They probably call that success.
-
@0xabad1dea Just before they did the ‘security’ stuff, someone sent me a ‘comprehensive code review’ of one of my projects. Most of the performance things were lifted directly from open issues I’d written, but one that it flagged as high importance was an intentional design decision.
It pointed out that one operation does a broadcast cache invalidation that invalidates all caches, rather than the subset that it actually needs to. The recommendation was to switch to having fine-grained cache state tracking. The problem is, that actually made cache accesses more expensive on the most performance-critical path in the system. Cache invalidations happened less than once every million accesses in typical programs, so this was an incredibly unusual case and optimising for it hurt performance everywhere else.
But, because it didn’t actually understand the code or the context, it saw a pattern that was slow and matched it to a high-severity warning.
I would expect it to do exactly the same thing with security reports. Memory-safety bug in a code path that can be reached only via trusted data? Highest severity!
@david_chisnall @0xabad1dea ugh, we have static tools that you can tune to be this annoying OpenText (this week, that’s who owns it anyway) Fortify’s default view is rich with the kind of false positives that the LLMs like to come up with.
You didn’t sanitize the command-line inputs! Sure, and if this were a suid executable that might matter, but it’s not.
Insecure randomness! This is a test function, doesn’t need to be cryptographically secure. -
J jwcph@helvede.net shared this topic
