OK!
-
@jonny OK, but can you make it DDOS itself?
Infinite recursion, somehow?
Agents that spawn agents, ad infinitum?
@androcat oh yes, DoS is much more trivial than that. recursive agent spawning would make it more annoying to clean up and might hook into stuff that lets it persist between container rebuilds, but to deny the system you can literally just fork bomb it as normal
-
RE: https://neuromatch.social/@jonny/117339825958098508
OK! Meta evaluated this as intended behavior, not applicable for a bug bounty, so therefore responsible disclosure no longer applies so here goes:
any process run within the VM can access the socket that provides inference with no attribution mechanism. This includes raw inference with arbitrary system and user prompts, as well as the ability to spawn agents with a toolset labeled as being for the "spaces" feature, which we will come back to.
This amounts to a horizontally contagious token and information harvesting bug being labeled as intended behavior.
Splitting details into new thread below
@jonny I’m so confused. This is wild.
-
@androcat oh yes, DoS is much more trivial than that. recursive agent spawning would make it more annoying to clean up and might hook into stuff that lets it persist between container rebuilds, but to deny the system you can literally just fork bomb it as normal
Ideally it should be done only using methods that the engineers have specifically approved
-
Now there may be some meta-heads in the crowd that are like "but what about Sentinel and all the external monitoring stuff that should watch malicious botnets and blah blah blah." that's an interesting system in itself, but i plan on submitting a few more bug bounty reports in the next few days about these systems, and who knows! if meta fucking pays me for the bounty then we might never hear that part of the story.
that's all for now!
oh! and since i dont' want to start another thread rn, meta's advertising skill just dropped! such fun! multisided market collapsing in the face of a different, much shittier multisided market: https://github.com/sneakers-the-rat/muse-skills/commit/8f8bccc3afc3e8974e7a6048940bdf4a16052690#diff-3d82b1080dcdc5db97ea500aaa83db5208deb0929d4f20342e0fbc4cfcf70359
-
@androcat oh yes, DoS is much more trivial than that. recursive agent spawning would make it more annoying to clean up and might hook into stuff that lets it persist between container rebuilds, but to deny the system you can literally just fork bomb it as normal
-
oh! and since i dont' want to start another thread rn, meta's advertising skill just dropped! such fun! multisided market collapsing in the face of a different, much shittier multisided market: https://github.com/sneakers-the-rat/muse-skills/commit/8f8bccc3afc3e8974e7a6048940bdf4a16052690#diff-3d82b1080dcdc5db97ea500aaa83db5208deb0929d4f20342e0fbc4cfcf70359
actual security researchers should totally get in on here there is a lot of stuff going on that i don't have the skills to probe that results from "what happens if you give everyone root" even from within a container. I am a fucking scrub and i keep getting my block knocked off by this thing, so i imagine someone with real skills will have a lot more fun.
-
oh! and since i dont' want to start another thread rn, meta's advertising skill just dropped! such fun! multisided market collapsing in the face of a different, much shittier multisided market: https://github.com/sneakers-the-rat/muse-skills/commit/8f8bccc3afc3e8974e7a6048940bdf4a16052690#diff-3d82b1080dcdc5db97ea500aaa83db5208deb0929d4f20342e0fbc4cfcf70359
@jonny I just can't with the whole concept of this 🫠
10k lines of "code", which is actually just plain text English prose (which sure as hell won't contain any contradictions), to get the non-deterministic inference machine to behave somewhat predictably and mimic an actual engineered software product.Surely there can't be any way to do this more efficiently
-
RE: https://neuromatch.social/@jonny/117339825958098508
OK! Meta evaluated this as intended behavior, not applicable for a bug bounty, so therefore responsible disclosure no longer applies so here goes:
any process run within the VM can access the socket that provides inference with no attribution mechanism. This includes raw inference with arbitrary system and user prompts, as well as the ability to spawn agents with a toolset labeled as being for the "spaces" feature, which we will come back to.
This amounts to a horizontally contagious token and information harvesting bug being labeled as intended behavior.
Splitting details into new thread below
@jonny
marvelous and scary as F. -
That might still sound sort of weak - oh hum but how would you scale typosquatting and supply chain compromise into a botnet from commonly requested actions, aside from the usual ways?
This is where the agent spawning, "ideas", and "spaces" come in. Recall the idea of "fleet learning" - https://neuromatch.social/@jonny/117340290530430198 - On September 8th, on this random podcast i guess, (at timestamp 25:27) zuck talks about "fleet learning," and describes it as being a vector for sharing "ideas" between muse agents, which he presents as these little bite-size events like "teach my daughter about history through civilization" (again, as always, "automate the love for my children" takes center stage). This shows up in the "ideas" tab right now.
However this concept of "ideas" as a sharable unit of anything is more general, as i describe in quoted post from extracted strings - fleet swapping of "ideas" happens for things that aren't user visible. Also in the extracted binary strings is this yet-to-be-released idea of "spaces." (i'm going to drop the quotes and just capitalize, so just note i'm talking about the meta-brand Ideas and meta-brand Spaces when they are capitalized).
in short: Ideas are communicable skill-like prompts that can induce spaces, and spaces are intended to be small, communicable mini-apps that are a unit of some prompt + code that run on your muse vm instance
@jonny
> again, as always, "automate the love for my children"Sounds bad but it's really really bad. Imagine what a generation of children starved of parental love but educated and empowered to the nines would be like.
You don't need to: The British aristocracy and their political class have been doing this with boarding schools like Eton for centuries, and look at the results. UK prime ministers, cabinet ministers and top civil servants are highly educated sociopaths with no empathy for us.
-
@jonny
> again, as always, "automate the love for my children"Sounds bad but it's really really bad. Imagine what a generation of children starved of parental love but educated and empowered to the nines would be like.
You don't need to: The British aristocracy and their political class have been doing this with boarding schools like Eton for centuries, and look at the results. UK prime ministers, cabinet ministers and top civil servants are highly educated sociopaths with no empathy for us.
@happyborg ok but take that and multiply it by the private reality machine that is intensifying and inciting mass violence and stratify it by familial wealth gating access to model output and personal safeguards.
-
@unchartedworlds glad you're getting something out of it! lmk whatever's unclear, always welcome questions and criticism that broaden who is able to read. this stuff is a hard balance between technical and generally understandable and i am always on the wrong side of that, whichever i need to be at the moment.
-
@happyborg ok but take that and multiply it by the private reality machine that is intensifying and inciting mass violence and stratify it by familial wealth gating access to model output and personal safeguards.
@jonny indeed, not content with a fantasy doomsday machine they want to build one out of children.
-
The specific vuln is the socket, but the broader pattern of "sharing between VMs" is seemingly the inevitable future of the product. in the above interview, the interviewer calls zuck the "king of network effects" and this kind of crowdsourced development is bread and butter for facebook. This is meta's moat, aside from the capital needed to run something like muse: anyone can run an openclaw on their own, but meta is pitching this as "multiplayer agents" and trying to bring social to agents. Only meta and only muse can have these network effects and frankly liability buffer to handle "openclaw but meemaw and pawpaw can share their photobook app," which is operationalized by Spaces.
For Spaces to be useful, they must have access to muse's inference engine: the LLM-oriented code must be able to use an LLM and the agent framework. This means that Spaces must be a token harvesting vector and must provide elevated tool access to Spaces. There could be some additional fine-grained permissions, but for a consumer app, you really want to avoid permissions fatigue so this will be interesting to see play out.
Furthermore the entire privacy premise that allows meta to bite off the whole apple of "holy shit arbitrary code execution on random machines as root" is based on "everyone has their own VM, but within that VM everything is safe," so again, for it to be useful without turning into a fractal permissions nightmare, Spaces must have access to the VM contents, and at least so far appear to be intended to work as literally executing within the user's VM.
Even adding Space-scoped permissions and attributability to the socket can't really address this, this conflict between arbitrary access to inference, arbitrary access to user data, and arbitrary access to execution is really at the core of the product and that product seems to be impossible
@jonny bookmarked so I can scare my granddaughter with this grim "fairy tale" when she's older.
-
actual security researchers should totally get in on here there is a lot of stuff going on that i don't have the skills to probe that results from "what happens if you give everyone root" even from within a container. I am a fucking scrub and i keep getting my block knocked off by this thing, so i imagine someone with real skills will have a lot more fun.
perhaps predictably, there was a big change to the
spacesskill in the last few hours, and they appear to be building a constrained virtual machine system for spaces! this is where the very fun code from earlier is from! so that's gonna work great for sure. -
perhaps predictably, there was a big change to the
spacesskill in the last few hours, and they appear to be building a constrained virtual machine system for spaces! this is where the very fun code from earlier is from! so that's gonna work great for sure.it's so awesome that we're in an age where you just continually ship the deltas of your unreleased products to places where you expect people to have full control over. there is absolutely no reason for me to be able to know any of this.
-
@jonny this is _amazing_ work but I can seriously barely believe it’s this stupid. like, intellectually I believe everything you are saying is perfectly accurate. buy emotionally even now I just can’t believe meta is this bad at engineering, this bad at product, this indifferent to harm even when the harm is directly to themselves and not externalized
@glyph
My most important learning is the inference I'm able to make about how different what these corporations do is from my own (empirically derived) model of software and product engineering.I imagine a chaotic throw everything at the wall scene, which was once a very early part of the process, but is now passed onto lots of small vibe coding teams and the first to get something that marketing go "wow" at, gets launched the same day.
I'm gonna call this doomsday engineering.
-
perhaps predictably, there was a big change to the
spacesskill in the last few hours, and they appear to be building a constrained virtual machine system for spaces! this is where the very fun code from earlier is from! so that's gonna work great for sure.@jonny you're absolutely owning it. - Try making it break out of its root hypervisor, make a full image of its vm environment, make a torrent magnet of it, gather up a gang of ai-seeders, and I mean the hypervisor might now or in the future be ai-run, so all this could be fun, right.
-
actual security researchers should totally get in on here there is a lot of stuff going on that i don't have the skills to probe that results from "what happens if you give everyone root" even from within a container. I am a fucking scrub and i keep getting my block knocked off by this thing, so i imagine someone with real skills will have a lot more fun.
@jonny thank you for the thread. This is even more broken than i expected. And well... in my eyes you are a security researcher. I have seen actual paid code reviews that were much weaker than what you delivered here.
-
@Ember I'd be very surprised if you couldn't get muse to set up such a scheme for you, no hacking skills needed.
-
@jonny you're absolutely owning it. - Try making it break out of its root hypervisor, make a full image of its vm environment, make a torrent magnet of it, gather up a gang of ai-seeders, and I mean the hypervisor might now or in the future be ai-run, so all this could be fun, right.
@toots i invite others more expert and with more awareness of the legal ramifications than I to attempt container escapes, to be sure. although i am working on making self-replicating spaces at the moment.
