I know Mastodon hates LLM's and AI.
-
@drwhax so disclaimer, I’m not a coder or understand much about coding, but I have to mention this because it did make me laugh (hope someone finds this funny too)
When you mentioned your “parrot” at work breaking things apart & finding “vulnerabilities” (yes I know you were referring to the program
) my first thought was : “if you could make a program like that I think the best name for it would be a Kia parrot” (use scientific name probably )
(From their propensity to break things apart)@em_and_future_cats I'm also a huge bird admirer so this is more fun
-
@MossyQuartz Apologies that this is just talking about computer vulnerabilities, but yes, you're right about risk modeling in the away from keyboard world!
@drwhax Computer vulnerabilities, alright that's fair. When I took the Computerized Accounting class in 1985, students were told the business should continue the manual accounting methods concurrent with the new processes for two years, or more, because certain daily functions could look alright while other annual functions might not be obvious until after the annual reports are printed and reviewed. That was how us old people were taught about computers, they said the data needed to be confirmed as accurate before we could trust it.
-
@atraidez I think you're missing the point of my post, your anger is right and it's only downhill from here on out. It's also not going away, while people will also getting hurt with it if we don't find ways to patch whole classes of vulnerabilities. It's a perfect storm
-
@drwhax Computer vulnerabilities, alright that's fair. When I took the Computerized Accounting class in 1985, students were told the business should continue the manual accounting methods concurrent with the new processes for two years, or more, because certain daily functions could look alright while other annual functions might not be obvious until after the annual reports are printed and reviewed. That was how us old people were taught about computers, they said the data needed to be confirmed as accurate before we could trust it.
@MossyQuartz I still think thats a good idea to this day
-
@drwhax I'm paid by Red Hat to do this, and am given access to all the resources I need -- but it must be soul destroying for an open source contributor just doing OSS for fun as a hobby.
I keep telling myself that all this security stuff isn't much fun, but it probably makes open source stronger in the long term.
I think it helps to have very strict rules upstream for the flood of AI reports, e.g. https://github.com/fwupd/fwupd/blob/main/SECURITY.md#reporting-a-vulnerability
@hughsie @drwhax I think a lot of projects aren't clear where they stand between professional and hobby development and how much responsibility they should be taking on. Just because you donated some work to the world as FOSS and a company picked it up and distributes it as part of a product, doesn't make you responsible for the "supply chain" your hobby project finds itself in. IMHO any company which ships FOSS (which is _all_ of them) needs to vendor their dependencies, or have a contractual agreement for maintenance, and not just rely on the goodwill of the original author or volunteers for things like security, QA / CI testing, etc. I feel like a number of devs have gotten suckered into a culture of emulating commercial professional development (using an issue tracker, reviewing merge/pull requests, having numbered version releases, using CI and test suites, etc.) for things which are fundamentally non-commercial hobby projects and end up deeply unhappy burning their nights and weekends doing software maintenance because "people depend on them", like a martyr, and it feels rude to tell those people to get stuffed.
I think a fundamental way we should structure human systems is to limit the responsibility (and correlated damage) any one person can have on their own because our poor little meat-brains aren't really built for it.
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax telling maintainers to run LLM based bug searches and start writing fixes, while not providing them with money, free time, or other required means and support, can be seen as a form of agression ...
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax Nah, i don't care anymore. I'd rather let them break everything than get blackmailed. Fuck it, honestly, the internet is a pile of shit these days anyway, let it burn. I'mm done feeling bad about this while everyone keeps holding down the pedal as we rush towards the wall.
What does it matter, seriously? Soon it will be too expensive for any normal person or project to run anyway and then only bad actors will use it. So what's this? A bandaid for the break dam? Pointless.
-
@jeroen94704 @drwhax I will probably end up doing exactly that, yes. Hope all of this fucking rots in hell.
@chroma @jeroen94704 @drwhax You'll then have to take refute as a mountain hermit as all the society is permeated by computers.
Computers with FOSS in them and hence, most certainly LLM'd material will be involved at some point.
Even without the software parts, processes and hardware (computer or not) will have the taint of the LLM too.
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax This may be one of the only niche use cases where LLM technology may actually be useful, and it may be what Linus Torvalds was talking about when it comes to AI and the Linux kernel. Directly deploying AI code in the kernel could violate the GPL.
But as for auditing existing code for potential vulnerabilities, and the pen-testing stuff, this may be a viable use case.
But that's not what the AI companies promote. They promote the AI-powered shovelware development use case instead.
-
@hughsie @drwhax I think a lot of projects aren't clear where they stand between professional and hobby development and how much responsibility they should be taking on. Just because you donated some work to the world as FOSS and a company picked it up and distributes it as part of a product, doesn't make you responsible for the "supply chain" your hobby project finds itself in. IMHO any company which ships FOSS (which is _all_ of them) needs to vendor their dependencies, or have a contractual agreement for maintenance, and not just rely on the goodwill of the original author or volunteers for things like security, QA / CI testing, etc. I feel like a number of devs have gotten suckered into a culture of emulating commercial professional development (using an issue tracker, reviewing merge/pull requests, having numbered version releases, using CI and test suites, etc.) for things which are fundamentally non-commercial hobby projects and end up deeply unhappy burning their nights and weekends doing software maintenance because "people depend on them", like a martyr, and it feels rude to tell those people to get stuffed.
I think a fundamental way we should structure human systems is to limit the responsibility (and correlated damage) any one person can have on their own because our poor little meat-brains aren't really built for it.
@raven667 @hughsie @drwhax EU is attempting parts of this. We'll see about how the realexisting version of this will turn out.
Other than that: Ensure awareness for said corporates by enforcing the existing rights on the components they use.
There'll probably have to be some sort of enforcement threshold or staggered requirements.And a large part of it is probably to just answer to requests for features/fixes/... with "pull request welcome" or "here's my hourly rate" to any corporate requests.
Unfortunately, this is not trained for. -
@buherator @drwhax Yup same. It's what we do at $dayjob to motivated customers, partially. I'm waiting for a cohort to realize that, if nobody does anything, nobody will look too bad.
And yup. It's all a bit fuked up. :3
@ljrk @buherator @drwhax Part of the source of this is the lawyers. In business, companies are typically shielded from liability if they follow "standard and customary business practices". So if "everyone else" is doing it that way, you can too without taking on the liability.
Of course, that takes you only so far. If your business is unable to function for two months, legal liability may well be the least of your concerns. But the problem with that threat is that it's a gray rhino risk. Those are really really hard to get businesses to spend on mitigating.
For better or worse, LLM-based attacks may remove most of the gray from those risks and reduce the resistance to mitigation.
-
@drwhax This may be one of the only niche use cases where LLM technology may actually be useful, and it may be what Linus Torvalds was talking about when it comes to AI and the Linux kernel. Directly deploying AI code in the kernel could violate the GPL.
But as for auditing existing code for potential vulnerabilities, and the pen-testing stuff, this may be a viable use case.
But that's not what the AI companies promote. They promote the AI-powered shovelware development use case instead.
@alice_pea_3526 the parrots are pretty good at pattern recognition which is where I think they shine. Some smaller projects as well, but architecture wise, you'll have to handhold them a lot. Its almost like having a junior that's good in some incredibly niche things, but you'll be iterating a lot over code.
I think curl maintainer said something similar, the reports used to be bad from LLM's and they suddenly got a lot better. It still requires a pair of human eyes to understand if its not hallucinating.
I wish it was better at eliminating whole bug classes tho, that'd be the ideal situation?
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax One thing I don't see anyone addressing is how we got into this situation in the first place, or how we get out of it, and I'm not talking about the LLM or any of the already well known and discussed downsides of that.
How did the software get so insecure in the first place? How do we prevent security vulnerabilities before they occur? Most of the risks aren't unknown or even novel, so why haven't we prioritized the solutions?
-
@drwhax One thing I don't see anyone addressing is how we got into this situation in the first place, or how we get out of it, and I'm not talking about the LLM or any of the already well known and discussed downsides of that.
How did the software get so insecure in the first place? How do we prevent security vulnerabilities before they occur? Most of the risks aren't unknown or even novel, so why haven't we prioritized the solutions?
@zimzat No one wants to fund it is what I think.
Suhosin did something like this for PHP, now it's just jvoisin maintaining snufflepagus: https://github.com/jvoisin/snuffleupagus
I don't remember if there was something similar for other languages. Grsecurity for the Linux Kernel, that's the three I know
-
I know Mastodon hates LLM's and AI. So here goes!
I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
Can't wait for the discussions on this!
@drwhax I think the biggest lesson here is that Blue Team can't rely on patching to keep everything safe.
We have to actually do the rest of the stuff that security calls for, and not just keep everything patched but do more to minimize the risks when one of these unsleeping tools gets turned on us.
And so many companies don't even have a proper inventory of what-all needs to be kept up to date.
This is going to suck so much.
-
@ljrk @buherator @drwhax Part of the source of this is the lawyers. In business, companies are typically shielded from liability if they follow "standard and customary business practices". So if "everyone else" is doing it that way, you can too without taking on the liability.
Of course, that takes you only so far. If your business is unable to function for two months, legal liability may well be the least of your concerns. But the problem with that threat is that it's a gray rhino risk. Those are really really hard to get businesses to spend on mitigating.
For better or worse, LLM-based attacks may remove most of the gray from those risks and reduce the resistance to mitigation.
@mweiss @buherator @drwhax Totally, I'm pretty sure that risks that "end/pause business" will be addressed soon-ish. However, risks that are... less so, e.g., data loss, data privacy stuff, ... we'll just get used to it. Or the many downtimes of GitHub.
-
@infosecdj @drwhax You're right in that it would help but good samaritans aren't enough to fix the problem. It would be best if there was put some actual funding into open-source security (given how important it is) but that's a political problem well out of our control as individual developers.
@png @drwhax That's true! In an ideal world that could be the norm. But until that's in place, at least some of us could chip in. Wasn't that the way things were ran in the good old days -- submitting patches for problems? A crowd pointing at problems all the time can be strongly demotivating, but it does not have to be that way. Yet I am still to hear of a security expert submitting solutions together with problems they point out. A rare animal indeed.
-
@drwhax perhaps the most efficient course of action is to let the evil AI do whatever they want, destroy all the software we rely on, (along with our electrical grids and fill the atmosphere with CO2) and cause the ultimate collapse of our technological society. Thus we will have fixed the vulnerabilities through obsolescence.
-
@elle ok doei
-
@chroma @jeroen94704 @drwhax You'll then have to take refute as a mountain hermit as all the society is permeated by computers.
Computers with FOSS in them and hence, most certainly LLM'd material will be involved at some point.
Even without the software parts, processes and hardware (computer or not) will have the taint of the LLM too.
@jti42 @jeroen94704 @drwhax yes, in the process. Thankfully, I saved money and invested to the point I don’t have to work anymore and I’m moving to my countryside house in Korea
