I know Mastodon hates LLM's and AI.
-
@drwhax
my point is that fixing vulns only works if your enemy finds the same vulns as you found@wolf480pl @drwhax
Often open source LLMs that you can run on your own computer find the same bugs with the appropriate tools and instructions. It's mostly hype based on fear or as Lee Vinsel named it: «criti-hype»
https://aisle.com/blog/ai-cybersecurity-after-mythos-the-jagged-frontier -
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax I've been very concerned about the emergence of Mythos-class AI that can weaponize vulns. I'm personally responding by adding formal memory safety proof coverage to critical libraries in C (notably: OpenSSL). Testing - which can identify many but not all bugs - I no longer consider sufficient. Instead we need formal verification to prove the *absence* of bugs/vulns. #aicoding #security
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax
If the imbalance between attack and defence is AI, why can't AI be used to make patches instead of exploits? -
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax Can't the LLMs also write patches to fix these vulnerabilities? If not, why?
-
@drwhax it just seems pretty optimistic to think anyone will have access to computers in ten years.
-
@drwhax
If the imbalance between attack and defence is AI, why can't AI be used to make patches instead of exploits?@koehntopp @drwhax The idea is to use the AI to defend against exploits found by AI. No matter what Anthropic will be paid, you can't break the tools of the oppressor with his tools.
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax
I'm glad you've invited discussions on this, because my immediate thoughts and questions diverge from the discussion I'm primarily seeing here. When you mention vulnerabilities, yes you mention they exist in code as well as in projects. When I think of vulnerabilities, I think of the method I am using to make and maintain a record of something or else to achieve a goal or state of physical safety. For example, if my household accounting spreadsheets and forecasts are handwritten then the vulnerability is risk of loss if the dog eats them, but risk is mitigated when I keep an offline digital file. When a family member starts a small business, I ask the questions learnt in my accounting and auditing classes because I believe the risks involved include thinking about how to prevent losses of whatever is important. I only had a couple of BASIC programming classes in 1984, but it seemed to me that in those days the instructor felt we should understand the process so that we could consider risks. I now wonder if the machine-assisted searching allows you to consider the assorted practical situations of each business whose vulnerabilities are being sought. -
@drwhax I've been very concerned about the emergence of Mythos-class AI that can weaponize vulns. I'm personally responding by adding formal memory safety proof coverage to critical libraries in C (notably: OpenSSL). Testing - which can identify many but not all bugs - I no longer consider sufficient. Instead we need formal verification to prove the *absence* of bugs/vulns. #aicoding #security
@davidfstr I liked Halvar Flake's analogy, which i was sometimes a long the lines of, a bug every 10 years instead of 1 year. I think we can do a lot to prevent it, but there's always going to be bugs. I also think Mythos is way too hyped and these capabilities already exist in modern frontier models.
-
@drwhax I am old enough to remember RAM prices fluctuating between "having to sell a kidney" and "oooh, I get to max out my motherboard if I collect the deposit on these empty beer bottles" several times.
-
@galacticstone Well, pandoras box is already open .. and we will never be able to close it sadly....
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax so disclaimer, I’m not a coder or understand much about coding, but I have to mention this because it did make me laugh (hope someone finds this funny too)
When you mentioned your “parrot” at work breaking things apart & finding “vulnerabilities” (yes I know you were referring to the program
) my first thought was : “if you could make a program like that I think the best name for it would be a Kia parrot” (use scientific name probably )
(From their propensity to break things apart) -
@drwhax
If the imbalance between attack and defence is AI, why can't AI be used to make patches instead of exploits?@koehntopp this parrot is really good at pattern recognition, it can code and fix some I think given enough constraints and a feedback loop of adversarial review, but it still might not be up to the style of the codebase or it might even introduce new vulnerabilities. I still think we also haven't it fully figured out yet? I think this needs more benchmarks that are reproducible in some fashion
-
@drwhax Can't the LLMs also write patches to fix these vulnerabilities? If not, why?
@fabrice I replied about this here: https://infosec.exchange/@drwhax/116998200727608092
-
@koehntopp this parrot is really good at pattern recognition, it can code and fix some I think given enough constraints and a feedback loop of adversarial review, but it still might not be up to the style of the codebase or it might even introduce new vulnerabilities. I still think we also haven't it fully figured out yet? I think this needs more benchmarks that are reproducible in some fashion
@drwhax
I'd prefer that to be the focus of research over breaking stuff faster -
@drwhax
I'm glad you've invited discussions on this, because my immediate thoughts and questions diverge from the discussion I'm primarily seeing here. When you mention vulnerabilities, yes you mention they exist in code as well as in projects. When I think of vulnerabilities, I think of the method I am using to make and maintain a record of something or else to achieve a goal or state of physical safety. For example, if my household accounting spreadsheets and forecasts are handwritten then the vulnerability is risk of loss if the dog eats them, but risk is mitigated when I keep an offline digital file. When a family member starts a small business, I ask the questions learnt in my accounting and auditing classes because I believe the risks involved include thinking about how to prevent losses of whatever is important. I only had a couple of BASIC programming classes in 1984, but it seemed to me that in those days the instructor felt we should understand the process so that we could consider risks. I now wonder if the machine-assisted searching allows you to consider the assorted practical situations of each business whose vulnerabilities are being sought.@MossyQuartz Apologies that this is just talking about computer vulnerabilities, but yes, you're right about risk modeling in the away from keyboard world!
-
@drwhax so disclaimer, I’m not a coder or understand much about coding, but I have to mention this because it did make me laugh (hope someone finds this funny too)
When you mentioned your “parrot” at work breaking things apart & finding “vulnerabilities” (yes I know you were referring to the program) my first thought was : “if you could make a program like that I think the best name for it would be a Kia parrot” (use scientific name probably )
(From their propensity to break things apart)@em_and_future_cats I'm also a huge bird admirer so this is more fun
-
@MossyQuartz Apologies that this is just talking about computer vulnerabilities, but yes, you're right about risk modeling in the away from keyboard world!
@drwhax Computer vulnerabilities, alright that's fair. When I took the Computerized Accounting class in 1985, students were told the business should continue the manual accounting methods concurrent with the new processes for two years, or more, because certain daily functions could look alright while other annual functions might not be obvious until after the annual reports are printed and reviewed. That was how us old people were taught about computers, they said the data needed to be confirmed as accurate before we could trust it.
-
@atraidez I think you're missing the point of my post, your anger is right and it's only downhill from here on out. It's also not going away, while people will also getting hurt with it if we don't find ways to patch whole classes of vulnerabilities. It's a perfect storm
-
@drwhax Computer vulnerabilities, alright that's fair. When I took the Computerized Accounting class in 1985, students were told the business should continue the manual accounting methods concurrent with the new processes for two years, or more, because certain daily functions could look alright while other annual functions might not be obvious until after the annual reports are printed and reviewed. That was how us old people were taught about computers, they said the data needed to be confirmed as accurate before we could trust it.
@MossyQuartz I still think thats a good idea to this day
-
@drwhax I'm paid by Red Hat to do this, and am given access to all the resources I need -- but it must be soul destroying for an open source contributor just doing OSS for fun as a hobby.
I keep telling myself that all this security stuff isn't much fun, but it probably makes open source stronger in the long term.
I think it helps to have very strict rules upstream for the flood of AI reports, e.g. https://github.com/fwupd/fwupd/blob/main/SECURITY.md#reporting-a-vulnerability
@hughsie @drwhax I think a lot of projects aren't clear where they stand between professional and hobby development and how much responsibility they should be taking on. Just because you donated some work to the world as FOSS and a company picked it up and distributes it as part of a product, doesn't make you responsible for the "supply chain" your hobby project finds itself in. IMHO any company which ships FOSS (which is _all_ of them) needs to vendor their dependencies, or have a contractual agreement for maintenance, and not just rely on the goodwill of the original author or volunteers for things like security, QA / CI testing, etc. I feel like a number of devs have gotten suckered into a culture of emulating commercial professional development (using an issue tracker, reviewing merge/pull requests, having numbered version releases, using CI and test suites, etc.) for things which are fundamentally non-commercial hobby projects and end up deeply unhappy burning their nights and weekends doing software maintenance because "people depend on them", like a martyr, and it feels rude to tell those people to get stuffed.
I think a fundamental way we should structure human systems is to limit the responsibility (and correlated damage) any one person can have on their own because our poor little meat-brains aren't really built for it.
