I know Mastodon hates LLM's and AI.
-
@drwhax I think we need a bit of nuance here.
As much as I am no fan of LLMs, I can see the value of running them over one's own codebase in order to find vulnerabilities.
But I remain unconvinced that vibe-coded fixes are a good idea, or ever will be. I think this needs to be clearly said.
The reason is that these are fundamentally different problems.
@drwhax finding vulnerabilities can be stochastic because it has a very clear and effective verification function: either the exploit works or it does not. Exploit code can be messy and convoluted, as it is not going to be maintained after the vulnerability is fixed.
Vibe-coding fixes does not have that kind of verification function: the fix must not only close the specific vulnerability, but *also* not introduce new ones or re-introduce old ones, and it has to be maintainable in the future.
-
@drwhax I think we need a bit of nuance here.
As much as I am no fan of LLMs, I can see the value of running them over one's own codebase in order to find vulnerabilities.
But I remain unconvinced that vibe-coded fixes are a good idea, or ever will be. I think this needs to be clearly said.
The reason is that these are fundamentally different problems.
@rysiek Human in the loop is important here, vibe-coded patches are most of the time not very good. However, the models do tend to improve, but what really helps in this case in lets say an C/C++ code base, alright parrot, make me an fuzzer that fuzzes these paths and we'll let it run on every commit so we can catch things.
Last night, well, eherm,... 2am, my parrot was adding a fuzzer to code that hadn't been fuzzed and it showed, bugs came falling out.
The human needs to stay in the loop with all these things of course, i'm not advocating for a pure AI/LLM/Parrot solution. But oh boy, are they good at pattern recognition.
-
@drwhax finding vulnerabilities can be stochastic because it has a very clear and effective verification function: either the exploit works or it does not. Exploit code can be messy and convoluted, as it is not going to be maintained after the vulnerability is fixed.
Vibe-coding fixes does not have that kind of verification function: the fix must not only close the specific vulnerability, but *also* not introduce new ones or re-introduce old ones, and it has to be maintainable in the future.
@drwhax and in the end this will come down to managing limited resources and incentives, as always.
If a company can spend $1m in tokens on their codebase, are they incentivized to spend it on finding vulnerabilities, or churning out new vibe-coded features? I know what my guess is.
-
@can yes for sure!
There's a couple things you can do, one of them is indeed as basic as, this is app XYZ, written in PHP, review the codebase for security problems, then adversarially review your findings before writing them down in findings/ per bug in markdown. Also make a root cause and look for variants of the same class of issues across the code base.
What are some good skills to run is from Trail of Bits which are open-source: https://github.com/trailofbits/skills
If it's C/C++, LLM's are pretty good at making fuzzers and seeding the dictionary needed to fuzz them effectively.
If it's open-source you can apply for some openai codex api grant: https://openai.com/form/codex-open-source-fund/
@drwhax thanks!
-
@drwhax and in the end this will come down to managing limited resources and incentives, as always.
If a company can spend $1m in tokens on their codebase, are they incentivized to spend it on finding vulnerabilities, or churning out new vibe-coded features? I know what my guess is.
@rysiek For sure and the other part is guardrails that currently aren't aligned with the maintainer from these parrots either. Patching is hard or it'll fail because "CYBER"
-
@rysiek Human in the loop is important here, vibe-coded patches are most of the time not very good. However, the models do tend to improve, but what really helps in this case in lets say an C/C++ code base, alright parrot, make me an fuzzer that fuzzes these paths and we'll let it run on every commit so we can catch things.
Last night, well, eherm,... 2am, my parrot was adding a fuzzer to code that hadn't been fuzzed and it showed, bugs came falling out.
The human needs to stay in the loop with all these things of course, i'm not advocating for a pure AI/LLM/Parrot solution. But oh boy, are they good at pattern recognition.
@drwhax yup. All I am saying is that this distinction is important and needs to be constantly stressed, because people will end up creating more problems by vibe-coding.
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax The bottleneck seems to be a lack of maintainers, so I don't really see how running LLMs on codebases will fix that. It'll take a maintainer to make the right decision based on LLM output and we're clearly running out of those
-
@drwhax The bottleneck seems to be a lack of maintainers, so I don't really see how running LLMs on codebases will fix that. It'll take a maintainer to make the right decision based on LLM output and we're clearly running out of those
@png sadly, you're right.
-
@drwhax and in the end this will come down to managing limited resources and incentives, as always.
If a company can spend $1m in tokens on their codebase, are they incentivized to spend it on finding vulnerabilities, or churning out new vibe-coded features? I know what my guess is.
@drwhax and just to drop some data on vibe-coded software quality:
https://www.faros.ai/blog/ai-acceleration-whiplash-takeawaysThe report tries to be very "AI"-positive. But the numbers don't lie: +861% code churn, +242% incidents per PR, +54% bugs per developer.
-
@drwhax Maybe the advice of the wiser among us will be heard after all these decades: focus on robust mitigations and attack surface reduction, because the moles can't be whacked anymore.
@buherator @drwhax I sometimes advise investors (poor life choices I guess) on how to deal with the increased risk of Mythos & others. And this was the bottom line.
I don't even need AI specifically to argue for that. The time-to-exploit has been dropping forever. We've built better catalogs and more versatile infrastructure and complex automations. It's very clear that this includes vuln research as well.
We had this already with static analysis hitting the field. We answered with "shift left": Building security into the dev process, since the earlier you do it, the less issues you'll end up with. Every step earlier has more downstream impact.
We've finally reached the stage where security as part of your design & architecture is not only best practice but becoming a hard requirement. Our automations (including LLMs) pushed us there and we need to deal with the outcome.
Or should. Because I'm a cynic and I've seen what's happened, I doubt that engineering will follow through with this. I see two paths and we'll move along both, and I don't like either: Acceptance and few centralized Oligopols.
1/2
-
@drwhax and just to drop some data on vibe-coded software quality:
https://www.faros.ai/blog/ai-acceleration-whiplash-takeawaysThe report tries to be very "AI"-positive. But the numbers don't lie: +861% code churn, +242% incidents per PR, +54% bugs per developer.
@rysiek human in the loop etc
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax i have personally helped 3 different fediverse server projects by sharing exploits with them that LLMs found. Most software is full of holes. Running an LLM to hunt for bugs is good advice. If you're not doing it, someone else will. -
@drwhax i have personally helped 3 different fediverse server projects by sharing exploits with them that LLMs found. Most software is full of holes. Running an LLM to hunt for bugs is good advice. If you're not doing it, someone else will.
@lain amen
-
@buherator @drwhax I sometimes advise investors (poor life choices I guess) on how to deal with the increased risk of Mythos & others. And this was the bottom line.
I don't even need AI specifically to argue for that. The time-to-exploit has been dropping forever. We've built better catalogs and more versatile infrastructure and complex automations. It's very clear that this includes vuln research as well.
We had this already with static analysis hitting the field. We answered with "shift left": Building security into the dev process, since the earlier you do it, the less issues you'll end up with. Every step earlier has more downstream impact.
We've finally reached the stage where security as part of your design & architecture is not only best practice but becoming a hard requirement. Our automations (including LLMs) pushed us there and we need to deal with the outcome.
Or should. Because I'm a cynic and I've seen what's happened, I doubt that engineering will follow through with this. I see two paths and we'll move along both, and I don't like either: Acceptance and few centralized Oligopols.
1/2
1. Acceptance. There's the saying "you just need to be more secure than your neighbor". It's not fully true but biz people hold it so. It's flip side is a cynic commentary on security, and a true one at that: If your neighbor is not secure, you don't need to be either. This is still true if you're getting pwned. When everyone's getting hacked regularly, society will accept this new reality. Only the defense sector will still care for security. We've seen similar developments in different industries: Fast fashion isn't an adequate replacement for garment that stands the test of time. But it's getting impossible to buy good clothes, even if you want to. But also with other goods: If everyone produces low quality items, it's financially more reasonable to do the same.
2. Centralization. Coz we won't do the right thing and reduce complexity, we need to find a way to make complexity "secure": Cloud, Big5, etc. Corps that have the resources to host an Exchange server securely... or in compliance with regulations. Every tool you do not let a big house host is a liability. Not coz they're necessarily more secure, but because nobody will blame you if something gets wrong. We had that with "buying IBM" once.
I hate this outlook but I seriously doubt that we will correct course.
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax Using the stones to destroy the stones seems an unavoidable task
-
@drwhax Using the stones to destroy the stones seems an unavoidable task
@decapitae Ideally we get more Suhosin's projects for all these different languages, but I doubt it'll happen at this point.
-
@rysiek human in the loop etc
@drwhax the problem is this works better for certain tasks (finding vulnerabilities) and much worse for other tasks (vibe-coding) because of the shape of these tasks.
"Human in the loop" is not the get-out-of-LLM-problems-free card people try to pretend it is.
Human in the loop works for vulnerability findings because there is a reliable way of verifying the finding. It clearly does not work well for vibe-coding at all because there is no such reliable way of verifying code correctness.
-
@drwhax the problem is this works better for certain tasks (finding vulnerabilities) and much worse for other tasks (vibe-coding) because of the shape of these tasks.
"Human in the loop" is not the get-out-of-LLM-problems-free card people try to pretend it is.
Human in the loop works for vulnerability findings because there is a reliable way of verifying the finding. It clearly does not work well for vibe-coding at all because there is no such reliable way of verifying code correctness.
@rysiek I think we'll get there in a number of years, the way the field is developing now we got all these super fast interconnects and HBM memory and not to mention advancements in the machine learning field.
I almost puke writing this lol
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax a lot of us at Red Hat have been working at 100% for the last 6 months on basically this. I personally spend ~2h/day on AISLE and Mythos reports and getting fixes upstream.
-
@can yes for sure!
There's a couple things you can do, one of them is indeed as basic as, this is app XYZ, written in PHP, review the codebase for security problems, then adversarially review your findings before writing them down in findings/ per bug in markdown. Also make a root cause and look for variants of the same class of issues across the code base.
What are some good skills to run is from Trail of Bits which are open-source: https://github.com/trailofbits/skills
If it's C/C++, LLM's are pretty good at making fuzzers and seeding the dictionary needed to fuzz them effectively.
If it's open-source you can apply for some openai codex api grant: https://openai.com/form/codex-open-source-fund/
