<![CDATA[I need people to understand that stuff like this will keep happening, for two reasons:]]>RE: https://cyberplace.social/@GossiTheDog/116676826944489315

I need people to understand that stuff like this will keep happening, for two reasons:

1. To be useful these chatbots need to have full access to everything they are supposed to "manage"; otherwise they are pointless.

2. Trying to stop prompt injection is basically trying to semantically filter natural language.

These tools have no model of the world, no ontology to anchor any "safety instructions" in. There will always be a way to talk one's way around them.

]]>https://forum.fedi.dk/topic/bbe0313d-9377-417e-b0e2-7dbbd79de363/i-need-people-to-understand-that-stuff-like-this-will-keep-happening-for-two-reasonsRSS for NodeSun, 14 Jun 2026 15:48:09 GMTTue, 02 Jun 2026 01:50:14 GMT60<![CDATA[Reply to I need people to understand that stuff like this will keep happening, for two reasons: on Tue, 02 Jun 2026 04:10:37 GMT]]>@rysiek I would assume that anything a chatbot has permission to do, will get done, given enough time. Instructions to an LLM are just text which can and will get ignored. Also the chatbot can say that they did something even though no action has taken place.

It's all just meaningless text to the LLM.

]]>https://forum.fedi.dk/post/https://mastodon.gamedev.place/users/dominikg/statuses/116678553584890189https://forum.fedi.dk/post/https://mastodon.gamedev.place/users/dominikg/statuses/116678553584890189Tue, 02 Jun 2026 04:10:37 GMT<![CDATA[Reply to I need people to understand that stuff like this will keep happening, for two reasons: on Tue, 02 Jun 2026 03:00:37 GMT]]>@rysiek @paco so you’re telling me they treat security as seriously as Meta treats privacy?

]]>https://forum.fedi.dk/post/https://mastodon.online/users/dgodon/statuses/116678278334254862https://forum.fedi.dk/post/https://mastodon.online/users/dgodon/statuses/116678278334254862Tue, 02 Jun 2026 03:00:37 GMT<![CDATA[Reply to I need people to understand that stuff like this will keep happening, for two reasons: on Tue, 02 Jun 2026 02:55:34 GMT]]>@rysiek

wait. so giving 4 year olds in the playground assault rifles can't ever be made safe? say it isn't so...

]]>https://forum.fedi.dk/post/https://infosec.exchange/users/paul_ipv6/statuses/116678258515907416https://forum.fedi.dk/post/https://infosec.exchange/users/paul_ipv6/statuses/116678258515907416Tue, 02 Jun 2026 02:55:34 GMT<![CDATA[Reply to I need people to understand that stuff like this will keep happening, for two reasons: on Tue, 02 Jun 2026 02:45:14 GMT]]>@paco @rysiek "putting security above all else" = "instructing the code bots to only write secure code." Then telling them again because they *really* mean it this time!

]]>https://forum.fedi.dk/post/https://mastodon.social/users/EdCates/statuses/116678217869009389https://forum.fedi.dk/post/https://mastodon.social/users/EdCates/statuses/116678217869009389Tue, 02 Jun 2026 02:45:14 GMT<![CDATA[Reply to I need people to understand that stuff like this will keep happening, for two reasons: on Tue, 02 Jun 2026 02:34:20 GMT]]>@rysiek “We are doubling down on this very important work, putting security above all else — before all other features and investments,” Nadella said before adding “at least for the rest of this week. Maybe even a whole month.” 😜

]]>https://forum.fedi.dk/post/https://infosec.exchange/users/paco/statuses/116678175024246176https://forum.fedi.dk/post/https://infosec.exchange/users/paco/statuses/116678175024246176Tue, 02 Jun 2026 02:34:20 GMT<![CDATA[Reply to I need people to understand that stuff like this will keep happening, for two reasons: on Tue, 02 Jun 2026 02:30:01 GMT]]>@paco Satya Nadella made sure Microsoft focused on security over 2 years ago, after all!
https://www.geekwire.com/2024/haunted-by-repeated-breaches-microsoft-is-putting-security-above-all-else-vows-ceo-satya-nadella/

]]>https://forum.fedi.dk/post/https://mstdn.social/users/rysiek/statuses/116678158040365874https://forum.fedi.dk/post/https://mstdn.social/users/rysiek/statuses/116678158040365874Tue, 02 Jun 2026 02:30:01 GMT<![CDATA[Reply to I need people to understand that stuff like this will keep happening, for two reasons: on Tue, 02 Jun 2026 02:27:44 GMT]]>@rysiek At the bottom of that article is a headline for suggested next article:
“Also read: Microsoft is making Teams secure by default, automatically enabling new protections to reduce AI-driven threats.”

It wasn’t secure by default? But they’re gonna change that?

And I love how it flip flops from rock solid certainty “secure by default” to corporate weasel-speak “reduce AI-driven threats” in the span of a single sentence.

]]>https://forum.fedi.dk/post/https://infosec.exchange/users/paco/statuses/116678149025087121https://forum.fedi.dk/post/https://infosec.exchange/users/paco/statuses/116678149025087121Tue, 02 Jun 2026 02:27:44 GMT<![CDATA[Reply to I need people to understand that stuff like this will keep happening, for two reasons: on Tue, 02 Jun 2026 02:11:26 GMT]]>@dancast oh yeah, they probably got generated by it, and in a way they always pass.

]]>https://forum.fedi.dk/post/https://mstdn.social/users/rysiek/statuses/116678084969695067https://forum.fedi.dk/post/https://mstdn.social/users/rysiek/statuses/116678084969695067Tue, 02 Jun 2026 02:11:26 GMT<![CDATA[Reply to I need people to understand that stuff like this will keep happening, for two reasons: on Tue, 02 Jun 2026 02:10:40 GMT]]>@rysiek I am sure it passed its unit tests.

]]>https://forum.fedi.dk/post/https://wandering.shop/users/dancast/statuses/116678081925855557https://forum.fedi.dk/post/https://wandering.shop/users/dancast/statuses/116678081925855557Tue, 02 Jun 2026 02:10:40 GMT<![CDATA[Reply to I need people to understand that stuff like this will keep happening, for two reasons: on Tue, 02 Jun 2026 02:10:26 GMT]]>@arichtman because surely there will be no way to prompt-inject a request to write a malicious python script and run it.

]]>https://forum.fedi.dk/post/https://mstdn.social/users/rysiek/statuses/116678080995906716https://forum.fedi.dk/post/https://mstdn.social/users/rysiek/statuses/116678080995906716Tue, 02 Jun 2026 02:10:26 GMT<![CDATA[Reply to I need people to understand that stuff like this will keep happening, for two reasons: on Tue, 02 Jun 2026 02:08:52 GMT]]>We are several years into this and the biggest companies peddling these tools still cannot figure out how to make their products not fall for advanced cyberattack techniques like *checks notes* asking nicely again.

Microslop Slopilot had (has?) a similar issue – Reprompt attack simply repeated the malicious prompt in a query parameter:
https://www.techrepublic.com/article/news-reprompt-attack-microsoft-copilot/

These are not going away.

]]>https://forum.fedi.dk/post/https://mstdn.social/users/rysiek/statuses/116678074862477010https://forum.fedi.dk/post/https://mstdn.social/users/rysiek/statuses/116678074862477010Tue, 02 Jun 2026 02:08:52 GMT<![CDATA[Reply to I need people to understand that stuff like this will keep happening, for two reasons: on Tue, 02 Jun 2026 02:00:10 GMT]]>One way out of this is compartmentalization, hard-limiting chatbot's access to certain resources. But that defeats the purpose of the chatbot – you can't have a chatbot that manages your mail without giving that chatbot access to your mail...

Another is to move towards more formalized instructions, which can then be properly constrained by permissions etc. But then you're re-inventing programming languages and access control, again defeating the purpose of a natural-language-processing chatbot.

]]>https://forum.fedi.dk/post/https://mstdn.social/users/rysiek/statuses/116678040636647436https://forum.fedi.dk/post/https://mstdn.social/users/rysiek/statuses/116678040636647436Tue, 02 Jun 2026 02:00:10 GMT<![CDATA[Reply to I need people to understand that stuff like this will keep happening, for two reasons: on Tue, 02 Jun 2026 01:55:53 GMT]]>@rysiek Big sorry 😅 — you are totally correct. I hadn't thought about that 🙏 xD

And yeah, their system is absolutely garbage from a security standpoint; they know it and still leave it like that… They could have disabled number visibility a long time ago and also could have made their AI garbage opt-in instead of enforced… And they still use the old double ratchet… And many, many, many more things.

]]>https://forum.fedi.dk/post/https://mas.to/users/nemo/statuses/116678023825299035https://forum.fedi.dk/post/https://mas.to/users/nemo/statuses/116678023825299035Tue, 02 Jun 2026 01:55:53 GMT<![CDATA[Reply to I need people to understand that stuff like this will keep happening, for two reasons: on Tue, 02 Jun 2026 01:54:32 GMT]]>@nemo I don't think blaming the victim here is the way to go. Meta created a hilariously insecure system, this is on them.

]]>https://forum.fedi.dk/post/https://mstdn.social/users/rysiek/statuses/116678018467228131https://forum.fedi.dk/post/https://mstdn.social/users/rysiek/statuses/116678018467228131Tue, 02 Jun 2026 01:54:32 GMT<![CDATA[Reply to I need people to understand that stuff like this will keep happening, for two reasons: on Tue, 02 Jun 2026 01:53:49 GMT]]>@rysiek Apparently it could have been prevented with a WhatsApp PIN. Why isn't Obama using a WhatsApp PIN or the strict account settings, which also enable a PIN by default? 🤷 Btw I don't use WhatsApp; still trying to learn about it because many people use it.

]]>https://forum.fedi.dk/post/https://mas.to/users/nemo/statuses/116678015696965592https://forum.fedi.dk/post/https://mas.to/users/nemo/statuses/116678015696965592Tue, 02 Jun 2026 01:53:49 GMT