Project Glasswing:
Claiming to have found 26 thousand real vulnerabilities but only 0.8% of them have resulted in a real fix in a real project after five months is dire. They blame it on the human independent review bottleneck, but human experts being paid for their time definitely have a higher throughput than that when working with data that’s actually actionable.
The assigned-at-Claude severity ratings are also dire. It assigns “high” or “critical” to 91% of findings. Most findings in the real world are low or medium. This should be especially true when using a magic machine to shake out every last little issue that was overlooked by humans focused on the biggest risks.
[Edit: I should be more careful and note that this figure is calculated only from findings which have received a second opinion from project maintainers, which is much higher than the 0.8% fixed rate but much less than the entire dataset, so there’s probably bias towards reviewing those with critical ratings first. However, the maintainers found the high/critical rate to be quite inflated.]
Together this implies it’s generating thousands of trivial or nonsensical findings and labeling them HIGH DANGER CRITICAL MUST FIX, and the human independent verifiers are sifting for the rare needle in this haystack worth passing on. This isn’t really an improvement over the high-noise automated scanners we already had
(This is a corporate blog of someone with their own vulnerability management services to sell, so apply an appropriate number of grains of salt to their analysis. Filter keywords: AI LLM Anthropic)