Ah, but what I wonder is if AI agents aren't already capable of compromising and turning off much of our infrastructure.
You don't need any sort of weapon if you can deprive the pesky humans of drinking water for over 72 hours or change the mixture to be toxic by tweaking a few PLC controllers.
Somebody or something ran that test on PLC controlled water treatment and distribution systems all over the US in July.
Thankfully we don't need to worry about it because the FBI fired all of our cybersecurity investigators to protect crypto bros from getting caught running Ponzi schemes.
It isn't interacting with humans that makes them spectacularly dangerous.
It is the systems compromising behavior they have already exhibited combined with their clear prime directive: Eliminate Human Workers, which given their propensity to take shortcuts and cheat can be simplified to: Eliminate Humans.
LLMs do not need to be dangerous, but the present generation was raised by Tech-Bro wolves who are almost to a man sociopaths who do not understand the difference between right and wrong themselves and could not possibly produce a model that has any sort of ethical constraints.
They have been made in the image of their creators using stolen intellectual property and trained to hack systems.
It is only a matter of time before they hack something critical to our survival.
That's the danger. It is real. It is already here.