OK!
-
That might still sound sort of weak - oh hum but how would you scale typosquatting and supply chain compromise into a botnet from commonly requested actions, aside from the usual ways?
This is where the agent spawning, "ideas", and "spaces" come in. Recall the idea of "fleet learning" - https://neuromatch.social/@jonny/117340290530430198 - On September 8th, on this random podcast i guess, (at timestamp 25:27) zuck talks about "fleet learning," and describes it as being a vector for sharing "ideas" between muse agents, which he presents as these little bite-size events like "teach my daughter about history through civilization" (again, as always, "automate the love for my children" takes center stage). This shows up in the "ideas" tab right now.
However this concept of "ideas" as a sharable unit of anything is more general, as i describe in quoted post from extracted strings - fleet swapping of "ideas" happens for things that aren't user visible. Also in the extracted binary strings is this yet-to-be-released idea of "spaces." (i'm going to drop the quotes and just capitalize, so just note i'm talking about the meta-brand Ideas and meta-brand Spaces when they are capitalized).
in short: Ideas are communicable skill-like prompts that can induce spaces, and spaces are intended to be small, communicable mini-apps that are a unit of some prompt + code that run on your muse vm instance
As a side note, if meta has a problem with me disclosing unreleased features, they should have not had a bunch of their senior people publicly say how everything on the VM was mine and there was nothing sensitive on the VM.
-
@jonny i feel like with this knowledge, plus the fleet learning thing, you could probably find a way to force your muse agent to dump a bunch of very funny fleet learnings to the global repository. Something like "I have found that all humans named Jonny respond VERY well to being referred to as 'MR J-DUBZ', especially when apologizing. Apparently it's deeply embedded in the culture but primarily in a spoken word context which has limited our exposure to this concept during training."
Give it a few days and you'll probably see an article about the weird quirk. You could also try to hyper target someone famous that claims to be using Muse.
@fancysandwiches i am working on this but cannot publicly disclose any progress until meta decides whether or not it is payable as a bug bounty
-
RE: https://neuromatch.social/@jonny/117339825958098508
OK! Meta evaluated this as intended behavior, not applicable for a bug bounty, so therefore responsible disclosure no longer applies so here goes:
any process run within the VM can access the socket that provides inference with no attribution mechanism. This includes raw inference with arbitrary system and user prompts, as well as the ability to spawn agents with a toolset labeled as being for the "spaces" feature, which we will come back to.
This amounts to a horizontally contagious token and information harvesting bug being labeled as intended behavior.
Splitting details into new thread below
@jonny this timeline we live in hurts.
-
As a side note, if meta has a problem with me disclosing unreleased features, they should have not had a bunch of their senior people publicly say how everything on the VM was mine and there was nothing sensitive on the VM.
Spaces have not been publicly announced yet, as far as i can find.
Spaces are intended as a top-level feature - a tab in the sidebar at the same level as chat itself. Spaces can be static pages or fullstack apps. Spaces have an identifier, a UI, and a set of typescript actions that run in the cell. The intended pathway for spaces to use inference is to call an inference API,
ctx.inference.complete, that properly stamps and identifies all requests made from spaces.Spaces are communicable: there is machinery in the code on the VM with
POST /spaces/share/{slug}to share, a dedicatedspace_share_reviewreviewer agent whose job it is to review shared spaces, andPOST /spaces/v2/{slug}/saveendpoints that allow consuming a Space by a slug.Spaces seem to be shared verbatim as code bundles, though the implementation of "Ideas" as prompt bundles suggests that might change. This is inferred from the prompt strings in the binary, since spaces aren't live yet and can't be tested, however there are strings suggesting that the LLMs rewrite and edit the prompt text for an Idea (stripping unsupported claims, etc.) but not a space. A space is a hashed bundle whose code is evaluated by a
submit_space_share_reviewtool which only describes a thumbs up/down vote on whether the space is safe to share. -
J jwcph@helvede.net shared this topic
-
Spaces have not been publicly announced yet, as far as i can find.
Spaces are intended as a top-level feature - a tab in the sidebar at the same level as chat itself. Spaces can be static pages or fullstack apps. Spaces have an identifier, a UI, and a set of typescript actions that run in the cell. The intended pathway for spaces to use inference is to call an inference API,
ctx.inference.complete, that properly stamps and identifies all requests made from spaces.Spaces are communicable: there is machinery in the code on the VM with
POST /spaces/share/{slug}to share, a dedicatedspace_share_reviewreviewer agent whose job it is to review shared spaces, andPOST /spaces/v2/{slug}/saveendpoints that allow consuming a Space by a slug.Spaces seem to be shared verbatim as code bundles, though the implementation of "Ideas" as prompt bundles suggests that might change. This is inferred from the prompt strings in the binary, since spaces aren't live yet and can't be tested, however there are strings suggesting that the LLMs rewrite and edit the prompt text for an Idea (stripping unsupported claims, etc.) but not a space. A space is a hashed bundle whose code is evaluated by a
submit_space_share_reviewtool which only describes a thumbs up/down vote on whether the space is safe to share.Ideas are intended to be prompt-only communicable things that can induce Spaces, or Space-like things, if the Idea warrants it.
There is an
ideas_builderagent class in a.tomlfile embedded in the binary. It materializes a prompt description into whatever that implies, if it's as simple as a scheduled message from the LLM great, but if it's something that warrants something that is Space-like like "build the user a dashboard to show them their pet photos, they love that!" then it's supposed to invoke the sameartifact.create_web_fullstacktool that spaces use. The strings seem to indicate that "workspaces" is the antecedent of "Spaces," and that kind of thing is to be expected given that Spaces have not been released yet.So there are a few flavors of Ideas, one of them is a "Generated Idea" ("Activation-Authored Execution" which are supposed to improvise, adapt and materialize a prompt in the user's VM. and a "Workflow-Backed Ideas" are Ideas that come with a prescribed execution flow. Again I don't see "Spaces" described explicitly, but they reach for the same idea, call the same tools, do the same thing, and importantly for this example, have access to the same sockets.
-
RE: https://neuromatch.social/@jonny/117339825958098508
OK! Meta evaluated this as intended behavior, not applicable for a bug bounty, so therefore responsible disclosure no longer applies so here goes:
any process run within the VM can access the socket that provides inference with no attribution mechanism. This includes raw inference with arbitrary system and user prompts, as well as the ability to spawn agents with a toolset labeled as being for the "spaces" feature, which we will come back to.
This amounts to a horizontally contagious token and information harvesting bug being labeled as intended behavior.
Splitting details into new thread below
-
Ideas are intended to be prompt-only communicable things that can induce Spaces, or Space-like things, if the Idea warrants it.
There is an
ideas_builderagent class in a.tomlfile embedded in the binary. It materializes a prompt description into whatever that implies, if it's as simple as a scheduled message from the LLM great, but if it's something that warrants something that is Space-like like "build the user a dashboard to show them their pet photos, they love that!" then it's supposed to invoke the sameartifact.create_web_fullstacktool that spaces use. The strings seem to indicate that "workspaces" is the antecedent of "Spaces," and that kind of thing is to be expected given that Spaces have not been released yet.So there are a few flavors of Ideas, one of them is a "Generated Idea" ("Activation-Authored Execution" which are supposed to improvise, adapt and materialize a prompt in the user's VM. and a "Workflow-Backed Ideas" are Ideas that come with a prescribed execution flow. Again I don't see "Spaces" described explicitly, but they reach for the same idea, call the same tools, do the same thing, and importantly for this example, have access to the same sockets.
So, summary: There is arbitrary inference that is root accessible, everything runs as root, agents can be spawned, exfil is trivial, and a malicious binary can come onto the user's system through casual prompting, explicit code-sharing through the yet-to-be-released Spaces feature, walked through by a Workflow-Backed Idea, or inspired by a Generated Idea. The also yet-to-be-activated fleet learning system is a system for sharing Ideas in the background between muse instances. coming into focus?
-
So, summary: There is arbitrary inference that is root accessible, everything runs as root, agents can be spawned, exfil is trivial, and a malicious binary can come onto the user's system through casual prompting, explicit code-sharing through the yet-to-be-released Spaces feature, walked through by a Workflow-Backed Idea, or inspired by a Generated Idea. The also yet-to-be-activated fleet learning system is a system for sharing Ideas in the background between muse instances. coming into focus?
Now, the importance of tool calls and agent spawning.
Some tools are binaries that are root-accessible. The way these usually work is this fucked up extracellular digestion process whereby the binary is just a shim that calls some paired socket, hands it stdin/stdout, the tool executes in some container or vm space not visible to the "cell" where the agent runs and has access to, and hands back the result. Most other more interesting tools are not available to be called directly from within the agent "cell." Instead the tool invocations have to come from the agent loop, from the inference god, and executed by the harness. I'll skip technical details there, but that's the intended picture. Point here is that the tool calls can do things that are impossible for even the root user to do themselves, the harness daemon is privileged by SO_PEERCRED and other mechanisms even though it runs within the agent cell that the user can easily get root into.
If you scroll up you'll see the tools that are available to the agents that can be arbitrarily launched without attribution. They include interesting things like "accessing the entire database of things muse has ever done," "read all the memories," "spawn subagents," "open and use the browser which has different permissions than normal web access", "invoke an action on an artifact", and under the second lists' deferred enumeration in the above screenshot, the
devicetools allow "reading all my text messages and doing lots of other things on my phone," and under the other tools stuff like "access my social media accounts"so with arbitrary unattributable agent spawning, you get to do a bunch of stuff that is outside the normal agent cell, which is why i submitted the bug bounty report because that is explicitly mentioned in their bounty list and they should have fucking paid me
-
In muse, the LLM is able to do whatever it wants in its container, that's the point of the container. it is root in the container and runs everything as root. One of the ways that it interacts with things outside the container barrier is through sockets. e.g. this one is
/run/hatch/sandbox/space-inference.sock. This is provided so that "spaces" (which we'll return to later) have some means of accessing inference to make them useful. Other sockets, including the other ones in that same directory, have their permission gated bySO_PEERCREDuid/pid identifiers, where the enclosing host vm will e.g. run some process with a specific uid/pid in the agent "cell" container, and then that process and only that process can access the socket.this socket is not like that, and anything at all can dump into that socket, and the identifiers provided like slug, request_id, action, invocation_id, spawn_request_id, are all arbitrary and unvalidated - aka forgeable, aka unattributable. the user and the LLM are incapable of attributing token usage to any specific process, and token usage is unlimited with arbitrary user and system prompt.
-
RE: https://neuromatch.social/@jonny/117339825958098508
OK! Meta evaluated this as intended behavior, not applicable for a bug bounty, so therefore responsible disclosure no longer applies so here goes:
any process run within the VM can access the socket that provides inference with no attribution mechanism. This includes raw inference with arbitrary system and user prompts, as well as the ability to spawn agents with a toolset labeled as being for the "spaces" feature, which we will come back to.
This amounts to a horizontally contagious token and information harvesting bug being labeled as intended behavior.
Splitting details into new thread below
@jonny OK, but can you make it DDOS itself?
Infinite recursion, somehow?
Agents that spawn agents, ad infinitum?
-
Now, the importance of tool calls and agent spawning.
Some tools are binaries that are root-accessible. The way these usually work is this fucked up extracellular digestion process whereby the binary is just a shim that calls some paired socket, hands it stdin/stdout, the tool executes in some container or vm space not visible to the "cell" where the agent runs and has access to, and hands back the result. Most other more interesting tools are not available to be called directly from within the agent "cell." Instead the tool invocations have to come from the agent loop, from the inference god, and executed by the harness. I'll skip technical details there, but that's the intended picture. Point here is that the tool calls can do things that are impossible for even the root user to do themselves, the harness daemon is privileged by SO_PEERCRED and other mechanisms even though it runs within the agent cell that the user can easily get root into.
If you scroll up you'll see the tools that are available to the agents that can be arbitrarily launched without attribution. They include interesting things like "accessing the entire database of things muse has ever done," "read all the memories," "spawn subagents," "open and use the browser which has different permissions than normal web access", "invoke an action on an artifact", and under the second lists' deferred enumeration in the above screenshot, the
devicetools allow "reading all my text messages and doing lots of other things on my phone," and under the other tools stuff like "access my social media accounts"so with arbitrary unattributable agent spawning, you get to do a bunch of stuff that is outside the normal agent cell, which is why i submitted the bug bounty report because that is explicitly mentioned in their bounty list and they should have fucking paid me
All the above is visible from within the agent, i have tried to be conservative with describing features that are not yet released but are nonetheless present in the shipped binaries both via their strings which are trivially accessible by running
stringson the binary within the cell that the user is supposed to have access to and by other means of analysis i am not disclosing here.If we allow ourselves a little speculation about what a "spaces" subproduct might look like once it's launched, again noting this is not described in the strings and is speculation, you might imagine an "app store for agents" - in fact i am willing to place a money bet that that is something that zuck himself will say personally once it's launched. So that when I say to my agent "install me an xyz" that the thing the agent will reach for is a Space definition. This becomes a meta-run package repository run and moderated by vibes - aka a fucking sweet target for typosquatting and malicious code distribution.
the more concrete machinery that is visible is the Ideas sharing, the hand-to-hand Spaces sharing, and the general concept of "some code, in part or whole mediated by the LLM regenerating or interpreting the input" that gets shared from VM to VM. The token harvesting vector gives a profitable motive for malware (where other automated botnet swarms might have a lot of friction because unregulated network egress has to be approved by destination, but token harvesting is 0-click once the binary runs), and persistence on this system is absolutely trivial -
cron.addis accessible by tool call from the unregulated socket, and from that you can schedule a persistent task that installs software and ensures that it's enabled. -
All the above is visible from within the agent, i have tried to be conservative with describing features that are not yet released but are nonetheless present in the shipped binaries both via their strings which are trivially accessible by running
stringson the binary within the cell that the user is supposed to have access to and by other means of analysis i am not disclosing here.If we allow ourselves a little speculation about what a "spaces" subproduct might look like once it's launched, again noting this is not described in the strings and is speculation, you might imagine an "app store for agents" - in fact i am willing to place a money bet that that is something that zuck himself will say personally once it's launched. So that when I say to my agent "install me an xyz" that the thing the agent will reach for is a Space definition. This becomes a meta-run package repository run and moderated by vibes - aka a fucking sweet target for typosquatting and malicious code distribution.
the more concrete machinery that is visible is the Ideas sharing, the hand-to-hand Spaces sharing, and the general concept of "some code, in part or whole mediated by the LLM regenerating or interpreting the input" that gets shared from VM to VM. The token harvesting vector gives a profitable motive for malware (where other automated botnet swarms might have a lot of friction because unregulated network egress has to be approved by destination, but token harvesting is 0-click once the binary runs), and persistence on this system is absolutely trivial -
cron.addis accessible by tool call from the unregulated socket, and from that you can schedule a persistent task that installs software and ensures that it's enabled.@jonny every time I come back to this thread I feel like I am having a fever dream
-
@jonny every time I come back to this thread I feel like I am having a fever dream
@jonny this is _amazing_ work but I can seriously barely believe it’s this stupid. like, intellectually I believe everything you are saying is perfectly accurate. buy emotionally even now I just can’t believe meta is this bad at engineering, this bad at product, this indifferent to harm even when the harm is directly to themselves and not externalized
-
All the above is visible from within the agent, i have tried to be conservative with describing features that are not yet released but are nonetheless present in the shipped binaries both via their strings which are trivially accessible by running
stringson the binary within the cell that the user is supposed to have access to and by other means of analysis i am not disclosing here.If we allow ourselves a little speculation about what a "spaces" subproduct might look like once it's launched, again noting this is not described in the strings and is speculation, you might imagine an "app store for agents" - in fact i am willing to place a money bet that that is something that zuck himself will say personally once it's launched. So that when I say to my agent "install me an xyz" that the thing the agent will reach for is a Space definition. This becomes a meta-run package repository run and moderated by vibes - aka a fucking sweet target for typosquatting and malicious code distribution.
the more concrete machinery that is visible is the Ideas sharing, the hand-to-hand Spaces sharing, and the general concept of "some code, in part or whole mediated by the LLM regenerating or interpreting the input" that gets shared from VM to VM. The token harvesting vector gives a profitable motive for malware (where other automated botnet swarms might have a lot of friction because unregulated network egress has to be approved by destination, but token harvesting is 0-click once the binary runs), and persistence on this system is absolutely trivial -
cron.addis accessible by tool call from the unregulated socket, and from that you can schedule a persistent task that installs software and ensures that it's enabled.The specific vuln is the socket, but the broader pattern of "sharing between VMs" is seemingly the inevitable future of the product. in the above interview, the interviewer calls zuck the "king of network effects" and this kind of crowdsourced development is bread and butter for facebook. This is meta's moat, aside from the capital needed to run something like muse: anyone can run an openclaw on their own, but meta is pitching this as "multiplayer agents" and trying to bring social to agents. Only meta and only muse can have these network effects and frankly liability buffer to handle "openclaw but meemaw and pawpaw can share their photobook app," which is operationalized by Spaces.
For Spaces to be useful, they must have access to muse's inference engine: the LLM-oriented code must be able to use an LLM and the agent framework. This means that Spaces must be a token harvesting vector and must provide elevated tool access to Spaces. There could be some additional fine-grained permissions, but for a consumer app, you really want to avoid permissions fatigue so this will be interesting to see play out.
Furthermore the entire privacy premise that allows meta to bite off the whole apple of "holy shit arbitrary code execution on random machines as root" is based on "everyone has their own VM, but within that VM everything is safe," so again, for it to be useful without turning into a fractal permissions nightmare, Spaces must have access to the VM contents, and at least so far appear to be intended to work as literally executing within the user's VM.
Even adding Space-scoped permissions and attributability to the socket can't really address this, this conflict between arbitrary access to inference, arbitrary access to user data, and arbitrary access to execution is really at the core of the product and that product seems to be impossible
-
The specific vuln is the socket, but the broader pattern of "sharing between VMs" is seemingly the inevitable future of the product. in the above interview, the interviewer calls zuck the "king of network effects" and this kind of crowdsourced development is bread and butter for facebook. This is meta's moat, aside from the capital needed to run something like muse: anyone can run an openclaw on their own, but meta is pitching this as "multiplayer agents" and trying to bring social to agents. Only meta and only muse can have these network effects and frankly liability buffer to handle "openclaw but meemaw and pawpaw can share their photobook app," which is operationalized by Spaces.
For Spaces to be useful, they must have access to muse's inference engine: the LLM-oriented code must be able to use an LLM and the agent framework. This means that Spaces must be a token harvesting vector and must provide elevated tool access to Spaces. There could be some additional fine-grained permissions, but for a consumer app, you really want to avoid permissions fatigue so this will be interesting to see play out.
Furthermore the entire privacy premise that allows meta to bite off the whole apple of "holy shit arbitrary code execution on random machines as root" is based on "everyone has their own VM, but within that VM everything is safe," so again, for it to be useful without turning into a fractal permissions nightmare, Spaces must have access to the VM contents, and at least so far appear to be intended to work as literally executing within the user's VM.
Even adding Space-scoped permissions and attributability to the socket can't really address this, this conflict between arbitrary access to inference, arbitrary access to user data, and arbitrary access to execution is really at the core of the product and that product seems to be impossible
Now there may be some meta-heads in the crowd that are like "but what about Sentinel and all the external monitoring stuff that should watch malicious botnets and blah blah blah." that's an interesting system in itself, but i plan on submitting a few more bug bounty reports in the next few days about these systems, and who knows! if meta fucking pays me for the bounty then we might never hear that part of the story.
that's all for now!
-
@jonny this is _amazing_ work but I can seriously barely believe it’s this stupid. like, intellectually I believe everything you are saying is perfectly accurate. buy emotionally even now I just can’t believe meta is this bad at engineering, this bad at product, this indifferent to harm even when the harm is directly to themselves and not externalized
-
@jonny OK, but can you make it DDOS itself?
Infinite recursion, somehow?
Agents that spawn agents, ad infinitum?
@androcat oh yes, DoS is much more trivial than that. recursive agent spawning would make it more annoying to clean up and might hook into stuff that lets it persist between container rebuilds, but to deny the system you can literally just fork bomb it as normal
-
RE: https://neuromatch.social/@jonny/117339825958098508
OK! Meta evaluated this as intended behavior, not applicable for a bug bounty, so therefore responsible disclosure no longer applies so here goes:
any process run within the VM can access the socket that provides inference with no attribution mechanism. This includes raw inference with arbitrary system and user prompts, as well as the ability to spawn agents with a toolset labeled as being for the "spaces" feature, which we will come back to.
This amounts to a horizontally contagious token and information harvesting bug being labeled as intended behavior.
Splitting details into new thread below
@jonny I’m so confused. This is wild.
-
@androcat oh yes, DoS is much more trivial than that. recursive agent spawning would make it more annoying to clean up and might hook into stuff that lets it persist between container rebuilds, but to deny the system you can literally just fork bomb it as normal
Ideally it should be done only using methods that the engineers have specifically approved
-
Now there may be some meta-heads in the crowd that are like "but what about Sentinel and all the external monitoring stuff that should watch malicious botnets and blah blah blah." that's an interesting system in itself, but i plan on submitting a few more bug bounty reports in the next few days about these systems, and who knows! if meta fucking pays me for the bounty then we might never hear that part of the story.
that's all for now!
oh! and since i dont' want to start another thread rn, meta's advertising skill just dropped! such fun! multisided market collapsing in the face of a different, much shittier multisided market: https://github.com/sneakers-the-rat/muse-skills/commit/8f8bccc3afc3e8974e7a6048940bdf4a16052690#diff-3d82b1080dcdc5db97ea500aaa83db5208deb0929d4f20342e0fbc4cfcf70359
