I know Mastodon hates LLM's and AI.
-
> I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure
You're missing the point. Even strong "AI critics" have already been seen to use these tools for these scans themselves. Just like other static code analysis tools and so on.
The main point and what people despise the most is when randoms use it and file bug reports they don't understand causing high workload validating their crap.
@agowa338 I agree and that's unfortunately the shitty side of it.
-
@drwhax I personally dont and will never vibe code because, in a very personal position, I love my brain and I like to think, and a brain without friction will cease to function (use it or lose it).
I love people making art (and yes, this includes code). And I deeply hate the contempt for people paying a tech bro to use genAI to dismiss and degrade those artists with a sort of vindictive glee, especially since most artists are already treated so badly. It is disgusting.
@drwhax But I don't mind the underlying tech as much as I dont have a beef against the, let's say Microsoft Kinect (a tool made in essence the same way).
So are you getting a Chinese open weight, AI (people won't say it, maybe because of fear of retaliation from the US, but most companies will do this) stuffing it in your server and having 90% effectiveness to find vulnerabilities and doing static analysis in code locally?
That sounds like an actual use case. Just one the tech bros don't like.
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax
do two differemt LLMs find the same bugs? -
@ljrk @drwhax Re: 1., the usual question we got from CISO and above on our reports is "how does it look compared to similar companies?". I agree that this is in part the "you don't have to outrun the bear" logic, but also that people in position don't want to look incompetent in front of their peers.
Another thing is that you don't get fired if you didn't follow the hackers advice, but you do get fired if you don't pass compliance which is one part BS, and the other part is easy to cheat.@buherator @drwhax Yup same. It's what we do at $dayjob to motivated customers, partially. I'm waiting for a cohort to realize that, if nobody does anything, nobody will look too bad.
And yup. It's all a bit fuked up. :3
-
@whvholst maybe things will turn cheap(er) again? I have some doubts about this and there's all kinds of ways to keep prices for components high by the companies and they've done this before and will do it again and they seem to get away with it mostly.
@drwhax I am old enough to remember RAM prices fluctuating between "having to sell a kidney" and "oooh, I get to max out my motherboard if I collect the deposit on these empty beer bottles" several times.
-
@drwhax
do two differemt LLMs find the same bugs?@wolf480pl sometimes, sometimes they don't, sometimes one finds a better way to chain vulnerabilities to achieve a certain objective. It all depends a bit on the harness as well, lots of small knobs to twist.
Some benchmarks are available on: https://exploitbench.ai/
-
@wolf480pl sometimes, sometimes they don't, sometimes one finds a better way to chain vulnerabilities to achieve a certain objective. It all depends a bit on the harness as well, lots of small knobs to twist.
Some benchmarks are available on: https://exploitbench.ai/
@drwhax
my point is that fixing vulns only works if your enemy finds the same vulns as you found -
@rysiek I think we'll get there in a number of years, the way the field is developing now we got all these super fast interconnects and HBM memory and not to mention advancements in the machine learning field.
I almost puke writing this lol
@drwhax I am very very doubtful we will in any meaningful way.
In the end coding is an exercise in translating intentions into machine-readable code, and also an exercise in communication between those whose intentions are enshrined in code, and those who then need to maintain it.
We will be able to automate some things slightly better, though. But then the question is: at what cost?
-
@drwhax I am very very doubtful we will in any meaningful way.
In the end coding is an exercise in translating intentions into machine-readable code, and also an exercise in communication between those whose intentions are enshrined in code, and those who then need to maintain it.
We will be able to automate some things slightly better, though. But then the question is: at what cost?
@rysiek everyone's mental capacity?
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax
Folk here hate AI probably because major players are evil, pirating copyright and exploiting privacy.
But this just like Google being evil doesn't mean search engine, email, and cloud drive are evil.There are open source LMs (Yes, open source, not only open weight) respecting copyright.
We should use tech to add value, not letting bad actors harm us with it. -
@rysiek everyone's mental capacity?
@drwhax that too, but I meant it even in the purely monetary sense of token costs.
-
@drwhax that too, but I meant it even in the purely monetary sense of token costs.
@rysiek once subsidy is gone, I don't think it makes a whole lot of sense, but, there might be advancements that tinnier models possible that are good enough at X or Y and then it's just hardware+electricity cost.
-
@fnrd you're right, but I think if we take this defeatist stance we're not going to improve things for the better.
Unfortunately, we'll not be able to make our own models, we're compute starved, our best bet is maybe open-weights models.
It's all a mess though, I do agree with that.
@drwhax I like to see it less as defeatist and more standing our ground. We don't need to rush. This will stop projects, hopefully before people burn out. That's the environment OpenAI and techbros have created. It's not given just because someone trashes your house you have to live there. You can build something new.
-
@rysiek once subsidy is gone, I don't think it makes a whole lot of sense, but, there might be advancements that tinnier models possible that are good enough at X or Y and then it's just hardware+electricity cost.
@drwhax oh I've been talking about smaller open-weights models for a long time now. A leaked Google memo ("we have no moat") mentioned them as a massive problem for them years ago. I have much less problem with using small, specialized, self-hosted, open-weights models.
In fact I know of at least one small company that already does this for vulnerability testing of their own code, avoiding most of the BS.
That said, it is still immensely expensive for them to run it on their own infra.
-
@drwhax oh I've been talking about smaller open-weights models for a long time now. A leaked Google memo ("we have no moat") mentioned them as a massive problem for them years ago. I have much less problem with using small, specialized, self-hosted, open-weights models.
In fact I know of at least one small company that already does this for vulnerability testing of their own code, avoiding most of the BS.
That said, it is still immensely expensive for them to run it on their own infra.
@drwhax in a way this is a question of how soon we finally get out of the Gartner hype cycle and people get to focus on figuring what these tools are *actually* useful for.
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax
This (which I do mostly agree with) is why LLMs feel like an attack on OSS and a massive centralisation push for control of computing, to me. They sell the attack and the defence. -
@png sadly, you're right.
-
The problem is that "we" is bearing a burden here that it can't actually sustain. It sounds great, but "we" does not include the adversaries, so "we" can't actually do this.
The "we" that includes people who think data centers are a bad idea can and should push back on them, just like we can and should push back on jet travel because of its carbon footprint. But if we unilaterally disarm, that makes us less, not more, likely to succeed.
Systemic problems can't be solved by opting out.
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax a lot of people have already said a lot of smart things in your replies, so i'll skip to a new direction:
why did you recently get access to these models? from a "media literacy" point of view, i feel like i don't know how to read your statement without knowing what your context is.
are you working with these models as part of a third-party, independent evaluation / audit? or are you working within a contract with either of these companies, do you receive money from them?
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax I don't want to debate, just one point: "and be creative enough".
Creative is the wrong word. It's not at all creative what they do.
It's pattern recognition, combining etc.
