I know Mastodon hates LLM's and AI.
-
@wolf480pl sometimes, sometimes they don't, sometimes one finds a better way to chain vulnerabilities to achieve a certain objective. It all depends a bit on the harness as well, lots of small knobs to twist.
Some benchmarks are available on: https://exploitbench.ai/
@drwhax
my point is that fixing vulns only works if your enemy finds the same vulns as you found -
@rysiek I think we'll get there in a number of years, the way the field is developing now we got all these super fast interconnects and HBM memory and not to mention advancements in the machine learning field.
I almost puke writing this lol
@drwhax I am very very doubtful we will in any meaningful way.
In the end coding is an exercise in translating intentions into machine-readable code, and also an exercise in communication between those whose intentions are enshrined in code, and those who then need to maintain it.
We will be able to automate some things slightly better, though. But then the question is: at what cost?
-
@drwhax I am very very doubtful we will in any meaningful way.
In the end coding is an exercise in translating intentions into machine-readable code, and also an exercise in communication between those whose intentions are enshrined in code, and those who then need to maintain it.
We will be able to automate some things slightly better, though. But then the question is: at what cost?
@rysiek everyone's mental capacity?
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax
Folk here hate AI probably because major players are evil, pirating copyright and exploiting privacy.
But this just like Google being evil doesn't mean search engine, email, and cloud drive are evil.There are open source LMs (Yes, open source, not only open weight) respecting copyright.
We should use tech to add value, not letting bad actors harm us with it. -
@rysiek everyone's mental capacity?
@drwhax that too, but I meant it even in the purely monetary sense of token costs.
-
@drwhax that too, but I meant it even in the purely monetary sense of token costs.
@rysiek once subsidy is gone, I don't think it makes a whole lot of sense, but, there might be advancements that tinnier models possible that are good enough at X or Y and then it's just hardware+electricity cost.
-
@fnrd you're right, but I think if we take this defeatist stance we're not going to improve things for the better.
Unfortunately, we'll not be able to make our own models, we're compute starved, our best bet is maybe open-weights models.
It's all a mess though, I do agree with that.
@drwhax I like to see it less as defeatist and more standing our ground. We don't need to rush. This will stop projects, hopefully before people burn out. That's the environment OpenAI and techbros have created. It's not given just because someone trashes your house you have to live there. You can build something new.
-
@rysiek once subsidy is gone, I don't think it makes a whole lot of sense, but, there might be advancements that tinnier models possible that are good enough at X or Y and then it's just hardware+electricity cost.
@drwhax oh I've been talking about smaller open-weights models for a long time now. A leaked Google memo ("we have no moat") mentioned them as a massive problem for them years ago. I have much less problem with using small, specialized, self-hosted, open-weights models.
In fact I know of at least one small company that already does this for vulnerability testing of their own code, avoiding most of the BS.
That said, it is still immensely expensive for them to run it on their own infra.
-
@drwhax oh I've been talking about smaller open-weights models for a long time now. A leaked Google memo ("we have no moat") mentioned them as a massive problem for them years ago. I have much less problem with using small, specialized, self-hosted, open-weights models.
In fact I know of at least one small company that already does this for vulnerability testing of their own code, avoiding most of the BS.
That said, it is still immensely expensive for them to run it on their own infra.
@drwhax in a way this is a question of how soon we finally get out of the Gartner hype cycle and people get to focus on figuring what these tools are *actually* useful for.
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax
This (which I do mostly agree with) is why LLMs feel like an attack on OSS and a massive centralisation push for control of computing, to me. They sell the attack and the defence. -
@png sadly, you're right.
-
The problem is that "we" is bearing a burden here that it can't actually sustain. It sounds great, but "we" does not include the adversaries, so "we" can't actually do this.
The "we" that includes people who think data centers are a bad idea can and should push back on them, just like we can and should push back on jet travel because of its carbon footprint. But if we unilaterally disarm, that makes us less, not more, likely to succeed.
Systemic problems can't be solved by opting out.
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax a lot of people have already said a lot of smart things in your replies, so i'll skip to a new direction:
why did you recently get access to these models? from a "media literacy" point of view, i feel like i don't know how to read your statement without knowing what your context is.
are you working with these models as part of a third-party, independent evaluation / audit? or are you working within a contract with either of these companies, do you receive money from them?
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax I don't want to debate, just one point: "and be creative enough".
Creative is the wrong word. It's not at all creative what they do.
It's pattern recognition, combining etc. -
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax not simply open source models, but inference chips that specialise on one model and car run them hundreds of times faster and cheaper, like the Taalas project
I honestly can't wait for the meltdown of my own sector. Imagine if making an internet service business suddenly becomes impossible, because if intelligent cyber attacks!
It's just so exciting, can't wait
-
@drwhax a lot of people have already said a lot of smart things in your replies, so i'll skip to a new direction:
why did you recently get access to these models? from a "media literacy" point of view, i feel like i don't know how to read your statement without knowing what your context is.
are you working with these models as part of a third-party, independent evaluation / audit? or are you working within a contract with either of these companies, do you receive money from them?
@catileptic No, I only pay a subscription myself and filled out two forms giving me access to a lessened guard rail LLM.
https://chatgpt.com/cyber?refresh_account=true
https://portal.anthropic.com/programs/cvp
I wanted access as at times I was hindered in reversing certain things on the non-cyber approved models. E.g forensic tooling
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax Yes. As code robots LLM’s should not be underestimated.
They lose to humans on depth, but beat us in breath, width and speed outright.
They are weapons. First and foremost they should be thought of as weapons.
-
@drwhax the problem is this works better for certain tasks (finding vulnerabilities) and much worse for other tasks (vibe-coding) because of the shape of these tasks.
"Human in the loop" is not the get-out-of-LLM-problems-free card people try to pretend it is.
Human in the loop works for vulnerability findings because there is a reliable way of verifying the finding. It clearly does not work well for vibe-coding at all because there is no such reliable way of verifying code correctness.
@rysiek @drwhax
"...the problem is this works better for certain tasks (finding vulnerabilities) and much worse for other tasks (vibe-coding)"This. Good at finding vulnerabilities, decent at creating an exploit blueprint, mediocre at verifying actual severity, and very much meh at creating a fix. Not the best combination out there, but alas, the first half necessitates the discussion.
-
@drwhax Yes. As code robots LLM’s should not be underestimated.
They lose to humans on depth, but beat us in breath, width and speed outright.
They are weapons. First and foremost they should be thought of as weapons.
@gimulnautti I think they already classify themselves as dual-use. which yes, I think is the right distinction
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax we also recently got the access and its insane. We are debating when the open source models getting close / similar to it to setup a cluster and provide resources to smaller open source projects and ngo security related folks to make sure these ones are not left behind. But this requires a lot of effort and funds and also training for people involved so for now its just on our list of possible projects.
