I know Mastodon hates LLM's and AI.
-
@drwhax the problem is this works better for certain tasks (finding vulnerabilities) and much worse for other tasks (vibe-coding) because of the shape of these tasks.
"Human in the loop" is not the get-out-of-LLM-problems-free card people try to pretend it is.
Human in the loop works for vulnerability findings because there is a reliable way of verifying the finding. It clearly does not work well for vibe-coding at all because there is no such reliable way of verifying code correctness.
@rysiek @drwhax
"...the problem is this works better for certain tasks (finding vulnerabilities) and much worse for other tasks (vibe-coding)"This. Good at finding vulnerabilities, decent at creating an exploit blueprint, mediocre at verifying actual severity, and very much meh at creating a fix. Not the best combination out there, but alas, the first half necessitates the discussion.
-
@drwhax Yes. As code robots LLM’s should not be underestimated.
They lose to humans on depth, but beat us in breath, width and speed outright.
They are weapons. First and foremost they should be thought of as weapons.
@gimulnautti I think they already classify themselves as dual-use. which yes, I think is the right distinction
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax we also recently got the access and its insane. We are debating when the open source models getting close / similar to it to setup a cluster and provide resources to smaller open source projects and ngo security related folks to make sure these ones are not left behind. But this requires a lot of effort and funds and also training for people involved so for now its just on our list of possible projects.
-
@catileptic No, I only pay a subscription myself and filled out two forms giving me access to a lessened guard rail LLM.
https://chatgpt.com/cyber?refresh_account=true
https://portal.anthropic.com/programs/cvp
I wanted access as at times I was hindered in reversing certain things on the non-cyber approved models. E.g forensic tooling
@drwhax @catileptic There should be a publicly funded project for a major foundation model, with both scientists AND humanists going over the training data with a fine comb.
Sadly, even the EU is too neoliberal and corporate to fund this (and always was).
We’re going need a new state.
-
@drwhax @catileptic There should be a publicly funded project for a major foundation model, with both scientists AND humanists going over the training data with a fine comb.
Sadly, even the EU is too neoliberal and corporate to fund this (and always was).
We’re going need a new state.
you're legitimizing the marketing stunts of these companies, which both aim, in the current media hype cycle, to make themselves the main coveted technology of armies around the world
all the while erasing responsibility and accountability
i want these companies held legally liable. for their involvement in bombing a school full of young girls, for example
i don't want to read an ex-Amnesty Security Lab employee praise the same company that contributed to war crimes
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax I think LLMs get the low hanging fruit of vulnerabilities, so you shouldn't trust an LLM's security audit as a definitive audit. As models get better, they'll catch higher up fruit, but still. -
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax Thanks for doing the work and sharing it.
On “Mastodon hates LLMs”: I think almost everyone has ambiguous feelings about generative AI and rightly so. Bizarrely, this includes the CEOs selling it to us.
The ones here who excommunicate you for sharing results that help the community ride out the storm… I hope are just a vocal minority
-
@infosecdj @drwhax You're right in that it would help but good samaritans aren't enough to fix the problem. It would be best if there was put some actual funding into open-source security (given how important it is) but that's a political problem well out of our control as individual developers.
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax What would the estimated cost be for these 7-9 hours, assuming no more subsidies? Just looking for a rough ball park figure, as almost everything I've heard up till this point is that while the LLMs are effective, the cost is significant enough that you could be paying someone to do the work directly.
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
the reality is that LLMs are incredibly powerful tools, but mastodon is so into reality denial
and while denying reality, the same people make fun of Trump supporters for denying reality (eg vaccines don't cause autism)be funny it wasn't so sad
-
@drwhax What would the estimated cost be for these 7-9 hours, assuming no more subsidies? Just looking for a rough ball park figure, as almost everything I've heard up till this point is that while the LLMs are effective, the cost is significant enough that you could be paying someone to do the work directly.
@macattackmicmac for that particular session it was roughly 122 USD in API costs, but since it was subsidized, it was only $20 in a subscription: https://gist.github.com/DrWhax/465da154f2ac1575cfc72aa71aa979e6
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax I would rather abandon an already dying field and pursue my dream of opening a coffee place rather than using LLMs for anything at all.
-
the reality is that LLMs are incredibly powerful tools, but mastodon is so into reality denial
and while denying reality, the same people make fun of Trump supporters for denying reality (eg vaccines don't cause autism)be funny it wasn't so sad
@failedLyndonLaRouchite They are, but they have flaws and the way these models are created is quite shit and there's a lot of things to say about that. It's not that I say, anthropic good, or openai good. It's more to say, we're entering an era with a lot of shit going to be thrown at people, companies and maintainers and I don't think people realize what's about to hit them!
-
@drwhax I would rather abandon an already dying field and pursue my dream of opening a coffee place rather than using LLMs for anything at all.
@chroma yeah i'd also like a farm!
-
@macattackmicmac for that particular session it was roughly 122 USD in API costs, but since it was subsidized, it was only $20 in a subscription: https://gist.github.com/DrWhax/465da154f2ac1575cfc72aa71aa979e6
@drwhax That is significantly cheaper that I was expecting.
-
@chroma yeah i'd also like a farm!
@drwhax And, to be honest, if any of the OSS/FOSS software I use dares to make ANYTHING with LLMs (not AI, but GenAI) especially relying on companies like Anthropic or OpenAI, that software goes to the trash bin.
I know is not rational, but one cannot hate these companies, and what the hell, the underlying technology has very few non-evil use cases.
-
@drwhax I think LLMs get the low hanging fruit of vulnerabilities, so you shouldn't trust an LLM's security audit as a definitive audit. As models get better, they'll catch higher up fruit, but still.
@edward I do think that's the case at the moment, cross-file vulnerabilities are sometimes not found. What I feel they're mostly good at is, pattern recognition, e.g, there's a specific bug class fixed from past git history that it found repeated or as a variant. After all, LLM's is just a stochastic parrot and it shows
-
@drwhax That is significantly cheaper that I was expecting.
@macattackmicmac yeah, it's pretty wild at the cost to be honest.
-
@rysiek @drwhax
"...the problem is this works better for certain tasks (finding vulnerabilities) and much worse for other tasks (vibe-coding)"This. Good at finding vulnerabilities, decent at creating an exploit blueprint, mediocre at verifying actual severity, and very much meh at creating a fix. Not the best combination out there, but alas, the first half necessitates the discussion.
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax What will any of this matter when we have turned the planet into a desert and caused our own extinction level event? There is an assumption that we will have access to electricity for any of this conversation to matter; when every day people have to deal with over 100° hot weather, rolling black outs, and drinking water restricted for anyone not rich/wealthy?
But let's keep protecting the people that made these problems and continue to make them worse. Justifying another technology built on theft and slave labor, I don't understand how anyone can be okay with that. We cannot separate the technology from its history, negative environmental impact, and that this technology is another scam to hoard even more wealth. The companies are using Enron's tactics for money and the packaging of debt into bad deals like 2008 in the USA is leading us to a very difficult time, where once again, these conversations will not matter as people will br trying to just survive.
