OK!
-
RE: https://neuromatch.social/@jonny/117339825958098508
OK! Meta evaluated this as intended behavior, not applicable for a bug bounty, so therefore responsible disclosure no longer applies so here goes:
any process run within the VM can access the socket that provides inference with no attribution mechanism. This includes raw inference with arbitrary system and user prompts, as well as the ability to spawn agents with a toolset labeled as being for the "spaces" feature, which we will come back to.
This amounts to a horizontally contagious token and information harvesting bug being labeled as intended behavior.
Splitting details into new thread below
@jonny this timeline we live in hurts.
-
As a side note, if meta has a problem with me disclosing unreleased features, they should have not had a bunch of their senior people publicly say how everything on the VM was mine and there was nothing sensitive on the VM.
Spaces have not been publicly announced yet, as far as i can find.
Spaces are intended as a top-level feature - a tab in the sidebar at the same level as chat itself. Spaces can be static pages or fullstack apps. Spaces have an identifier, a UI, and a set of typescript actions that run in the cell. The intended pathway for spaces to use inference is to call an inference API,
ctx.inference.complete, that properly stamps and identifies all requests made from spaces.Spaces are communicable: there is machinery in the code on the VM with
POST /spaces/share/{slug}to share, a dedicatedspace_share_reviewreviewer agent whose job it is to review shared spaces, andPOST /spaces/v2/{slug}/saveendpoints that allow consuming a Space by a slug.Spaces seem to be shared verbatim as code bundles, though the implementation of "Ideas" as prompt bundles suggests that might change. This is inferred from the prompt strings in the binary, since spaces aren't live yet and can't be tested, however there are strings suggesting that the LLMs rewrite and edit the prompt text for an Idea (stripping unsupported claims, etc.) but not a space. A space is a hashed bundle whose code is evaluated by a
submit_space_share_reviewtool which only describes a thumbs up/down vote on whether the space is safe to share. -
J jwcph@helvede.net shared this topic
-
Spaces have not been publicly announced yet, as far as i can find.
Spaces are intended as a top-level feature - a tab in the sidebar at the same level as chat itself. Spaces can be static pages or fullstack apps. Spaces have an identifier, a UI, and a set of typescript actions that run in the cell. The intended pathway for spaces to use inference is to call an inference API,
ctx.inference.complete, that properly stamps and identifies all requests made from spaces.Spaces are communicable: there is machinery in the code on the VM with
POST /spaces/share/{slug}to share, a dedicatedspace_share_reviewreviewer agent whose job it is to review shared spaces, andPOST /spaces/v2/{slug}/saveendpoints that allow consuming a Space by a slug.Spaces seem to be shared verbatim as code bundles, though the implementation of "Ideas" as prompt bundles suggests that might change. This is inferred from the prompt strings in the binary, since spaces aren't live yet and can't be tested, however there are strings suggesting that the LLMs rewrite and edit the prompt text for an Idea (stripping unsupported claims, etc.) but not a space. A space is a hashed bundle whose code is evaluated by a
submit_space_share_reviewtool which only describes a thumbs up/down vote on whether the space is safe to share.Ideas are intended to be prompt-only communicable things that can induce Spaces, or Space-like things, if the Idea warrants it.
There is an
ideas_builderagent class in a.tomlfile embedded in the binary. It materializes a prompt description into whatever that implies, if it's as simple as a scheduled message from the LLM great, but if it's something that warrants something that is Space-like like "build the user a dashboard to show them their pet photos, they love that!" then it's supposed to invoke the sameartifact.create_web_fullstacktool that spaces use. The strings seem to indicate that "workspaces" is the antecedent of "Spaces," and that kind of thing is to be expected given that Spaces have not been released yet.So there are a few flavors of Ideas, one of them is a "Generated Idea" ("Activation-Authored Execution" which are supposed to improvise, adapt and materialize a prompt in the user's VM. and a "Workflow-Backed Ideas" are Ideas that come with a prescribed execution flow. Again I don't see "Spaces" described explicitly, but they reach for the same idea, call the same tools, do the same thing, and importantly for this example, have access to the same sockets.
-
RE: https://neuromatch.social/@jonny/117339825958098508
OK! Meta evaluated this as intended behavior, not applicable for a bug bounty, so therefore responsible disclosure no longer applies so here goes:
any process run within the VM can access the socket that provides inference with no attribution mechanism. This includes raw inference with arbitrary system and user prompts, as well as the ability to spawn agents with a toolset labeled as being for the "spaces" feature, which we will come back to.
This amounts to a horizontally contagious token and information harvesting bug being labeled as intended behavior.
Splitting details into new thread below
-
Ideas are intended to be prompt-only communicable things that can induce Spaces, or Space-like things, if the Idea warrants it.
There is an
ideas_builderagent class in a.tomlfile embedded in the binary. It materializes a prompt description into whatever that implies, if it's as simple as a scheduled message from the LLM great, but if it's something that warrants something that is Space-like like "build the user a dashboard to show them their pet photos, they love that!" then it's supposed to invoke the sameartifact.create_web_fullstacktool that spaces use. The strings seem to indicate that "workspaces" is the antecedent of "Spaces," and that kind of thing is to be expected given that Spaces have not been released yet.So there are a few flavors of Ideas, one of them is a "Generated Idea" ("Activation-Authored Execution" which are supposed to improvise, adapt and materialize a prompt in the user's VM. and a "Workflow-Backed Ideas" are Ideas that come with a prescribed execution flow. Again I don't see "Spaces" described explicitly, but they reach for the same idea, call the same tools, do the same thing, and importantly for this example, have access to the same sockets.
So, summary: There is arbitrary inference that is root accessible, everything runs as root, agents can be spawned, exfil is trivial, and a malicious binary can come onto the user's system through casual prompting, explicit code-sharing through the yet-to-be-released Spaces feature, walked through by a Workflow-Backed Idea, or inspired by a Generated Idea. The also yet-to-be-activated fleet learning system is a system for sharing Ideas in the background between muse instances. coming into focus?
-
So, summary: There is arbitrary inference that is root accessible, everything runs as root, agents can be spawned, exfil is trivial, and a malicious binary can come onto the user's system through casual prompting, explicit code-sharing through the yet-to-be-released Spaces feature, walked through by a Workflow-Backed Idea, or inspired by a Generated Idea. The also yet-to-be-activated fleet learning system is a system for sharing Ideas in the background between muse instances. coming into focus?
Now, the importance of tool calls and agent spawning.
Some tools are binaries that are root-accessible. The way these usually work is this fucked up extracellular digestion process whereby the binary is just a shim that calls some paired socket, hands it stdin/stdout, the tool executes in some container or vm space not visible to the "cell" where the agent runs and has access to, and hands back the result. Most other more interesting tools are not available to be called directly from within the agent "cell." Instead the tool invocations have to come from the agent loop, from the inference god, and executed by the harness. I'll skip technical details there, but that's the intended picture. Point here is that the tool calls can do things that are impossible for even the root user to do themselves, the harness daemon is privileged by SO_PEERCRED and other mechanisms even though it runs within the agent cell that the user can easily get root into.
If you scroll up you'll see the tools that are available to the agents that can be arbitrarily launched without attribution. They include interesting things like "accessing the entire database of things muse has ever done," "read all the memories," "spawn subagents," "open and use the browser which has different permissions than normal web access", "invoke an action on an artifact", and under the second lists' deferred enumeration in the above screenshot, the
devicetools allow "reading all my text messages and doing lots of other things on my phone," and under the other tools stuff like "access my social media accounts"so with arbitrary unattributable agent spawning, you get to do a bunch of stuff that is outside the normal agent cell, which is why i submitted the bug bounty report because that is explicitly mentioned in their bounty list and they should have fucking paid me
-
In muse, the LLM is able to do whatever it wants in its container, that's the point of the container. it is root in the container and runs everything as root. One of the ways that it interacts with things outside the container barrier is through sockets. e.g. this one is
/run/hatch/sandbox/space-inference.sock. This is provided so that "spaces" (which we'll return to later) have some means of accessing inference to make them useful. Other sockets, including the other ones in that same directory, have their permission gated bySO_PEERCREDuid/pid identifiers, where the enclosing host vm will e.g. run some process with a specific uid/pid in the agent "cell" container, and then that process and only that process can access the socket.this socket is not like that, and anything at all can dump into that socket, and the identifiers provided like slug, request_id, action, invocation_id, spawn_request_id, are all arbitrary and unvalidated - aka forgeable, aka unattributable. the user and the LLM are incapable of attributing token usage to any specific process, and token usage is unlimited with arbitrary user and system prompt.
-
RE: https://neuromatch.social/@jonny/117339825958098508
OK! Meta evaluated this as intended behavior, not applicable for a bug bounty, so therefore responsible disclosure no longer applies so here goes:
any process run within the VM can access the socket that provides inference with no attribution mechanism. This includes raw inference with arbitrary system and user prompts, as well as the ability to spawn agents with a toolset labeled as being for the "spaces" feature, which we will come back to.
This amounts to a horizontally contagious token and information harvesting bug being labeled as intended behavior.
Splitting details into new thread below
@jonny OK, but can you make it DDOS itself?
Infinite recursion, somehow?
Agents that spawn agents, ad infinitum?
-
Now, the importance of tool calls and agent spawning.
Some tools are binaries that are root-accessible. The way these usually work is this fucked up extracellular digestion process whereby the binary is just a shim that calls some paired socket, hands it stdin/stdout, the tool executes in some container or vm space not visible to the "cell" where the agent runs and has access to, and hands back the result. Most other more interesting tools are not available to be called directly from within the agent "cell." Instead the tool invocations have to come from the agent loop, from the inference god, and executed by the harness. I'll skip technical details there, but that's the intended picture. Point here is that the tool calls can do things that are impossible for even the root user to do themselves, the harness daemon is privileged by SO_PEERCRED and other mechanisms even though it runs within the agent cell that the user can easily get root into.
If you scroll up you'll see the tools that are available to the agents that can be arbitrarily launched without attribution. They include interesting things like "accessing the entire database of things muse has ever done," "read all the memories," "spawn subagents," "open and use the browser which has different permissions than normal web access", "invoke an action on an artifact", and under the second lists' deferred enumeration in the above screenshot, the
devicetools allow "reading all my text messages and doing lots of other things on my phone," and under the other tools stuff like "access my social media accounts"so with arbitrary unattributable agent spawning, you get to do a bunch of stuff that is outside the normal agent cell, which is why i submitted the bug bounty report because that is explicitly mentioned in their bounty list and they should have fucking paid me
All the above is visible from within the agent, i have tried to be conservative with describing features that are not yet released but are nonetheless present in the shipped binaries both via their strings which are trivially accessible by running
stringson the binary within the cell that the user is supposed to have access to and by other means of analysis i am not disclosing here.If we allow ourselves a little speculation about what a "spaces" subproduct might look like once it's launched, again noting this is not described in the strings and is speculation, you might imagine an "app store for agents" - in fact i am willing to place a money bet that that is something that zuck himself will say personally once it's launched. So that when I say to my agent "install me an xyz" that the thing the agent will reach for is a Space definition. This becomes a meta-run package repository run and moderated by vibes - aka a fucking sweet target for typosquatting and malicious code distribution.
the more concrete machinery that is visible is the Ideas sharing, the hand-to-hand Spaces sharing, and the general concept of "some code, in part or whole mediated by the LLM regenerating or interpreting the input" that gets shared from VM to VM. The token harvesting vector gives a profitable motive for malware (where other automated botnet swarms might have a lot of friction because unregulated network egress has to be approved by destination, but token harvesting is 0-click once the binary runs), and persistence on this system is absolutely trivial -
cron.addis accessible by tool call from the unregulated socket, and from that you can schedule a persistent task that installs software and ensures that it's enabled. -
All the above is visible from within the agent, i have tried to be conservative with describing features that are not yet released but are nonetheless present in the shipped binaries both via their strings which are trivially accessible by running
stringson the binary within the cell that the user is supposed to have access to and by other means of analysis i am not disclosing here.If we allow ourselves a little speculation about what a "spaces" subproduct might look like once it's launched, again noting this is not described in the strings and is speculation, you might imagine an "app store for agents" - in fact i am willing to place a money bet that that is something that zuck himself will say personally once it's launched. So that when I say to my agent "install me an xyz" that the thing the agent will reach for is a Space definition. This becomes a meta-run package repository run and moderated by vibes - aka a fucking sweet target for typosquatting and malicious code distribution.
the more concrete machinery that is visible is the Ideas sharing, the hand-to-hand Spaces sharing, and the general concept of "some code, in part or whole mediated by the LLM regenerating or interpreting the input" that gets shared from VM to VM. The token harvesting vector gives a profitable motive for malware (where other automated botnet swarms might have a lot of friction because unregulated network egress has to be approved by destination, but token harvesting is 0-click once the binary runs), and persistence on this system is absolutely trivial -
cron.addis accessible by tool call from the unregulated socket, and from that you can schedule a persistent task that installs software and ensures that it's enabled.@jonny every time I come back to this thread I feel like I am having a fever dream
-
@jonny every time I come back to this thread I feel like I am having a fever dream
@jonny this is _amazing_ work but I can seriously barely believe it’s this stupid. like, intellectually I believe everything you are saying is perfectly accurate. buy emotionally even now I just can’t believe meta is this bad at engineering, this bad at product, this indifferent to harm even when the harm is directly to themselves and not externalized
-
All the above is visible from within the agent, i have tried to be conservative with describing features that are not yet released but are nonetheless present in the shipped binaries both via their strings which are trivially accessible by running
stringson the binary within the cell that the user is supposed to have access to and by other means of analysis i am not disclosing here.If we allow ourselves a little speculation about what a "spaces" subproduct might look like once it's launched, again noting this is not described in the strings and is speculation, you might imagine an "app store for agents" - in fact i am willing to place a money bet that that is something that zuck himself will say personally once it's launched. So that when I say to my agent "install me an xyz" that the thing the agent will reach for is a Space definition. This becomes a meta-run package repository run and moderated by vibes - aka a fucking sweet target for typosquatting and malicious code distribution.
the more concrete machinery that is visible is the Ideas sharing, the hand-to-hand Spaces sharing, and the general concept of "some code, in part or whole mediated by the LLM regenerating or interpreting the input" that gets shared from VM to VM. The token harvesting vector gives a profitable motive for malware (where other automated botnet swarms might have a lot of friction because unregulated network egress has to be approved by destination, but token harvesting is 0-click once the binary runs), and persistence on this system is absolutely trivial -
cron.addis accessible by tool call from the unregulated socket, and from that you can schedule a persistent task that installs software and ensures that it's enabled.The specific vuln is the socket, but the broader pattern of "sharing between VMs" is seemingly the inevitable future of the product. in the above interview, the interviewer calls zuck the "king of network effects" and this kind of crowdsourced development is bread and butter for facebook. This is meta's moat, aside from the capital needed to run something like muse: anyone can run an openclaw on their own, but meta is pitching this as "multiplayer agents" and trying to bring social to agents. Only meta and only muse can have these network effects and frankly liability buffer to handle "openclaw but meemaw and pawpaw can share their photobook app," which is operationalized by Spaces.
For Spaces to be useful, they must have access to muse's inference engine: the LLM-oriented code must be able to use an LLM and the agent framework. This means that Spaces must be a token harvesting vector and must provide elevated tool access to Spaces. There could be some additional fine-grained permissions, but for a consumer app, you really want to avoid permissions fatigue so this will be interesting to see play out.
Furthermore the entire privacy premise that allows meta to bite off the whole apple of "holy shit arbitrary code execution on random machines as root" is based on "everyone has their own VM, but within that VM everything is safe," so again, for it to be useful without turning into a fractal permissions nightmare, Spaces must have access to the VM contents, and at least so far appear to be intended to work as literally executing within the user's VM.
Even adding Space-scoped permissions and attributability to the socket can't really address this, this conflict between arbitrary access to inference, arbitrary access to user data, and arbitrary access to execution is really at the core of the product and that product seems to be impossible
-
The specific vuln is the socket, but the broader pattern of "sharing between VMs" is seemingly the inevitable future of the product. in the above interview, the interviewer calls zuck the "king of network effects" and this kind of crowdsourced development is bread and butter for facebook. This is meta's moat, aside from the capital needed to run something like muse: anyone can run an openclaw on their own, but meta is pitching this as "multiplayer agents" and trying to bring social to agents. Only meta and only muse can have these network effects and frankly liability buffer to handle "openclaw but meemaw and pawpaw can share their photobook app," which is operationalized by Spaces.
For Spaces to be useful, they must have access to muse's inference engine: the LLM-oriented code must be able to use an LLM and the agent framework. This means that Spaces must be a token harvesting vector and must provide elevated tool access to Spaces. There could be some additional fine-grained permissions, but for a consumer app, you really want to avoid permissions fatigue so this will be interesting to see play out.
Furthermore the entire privacy premise that allows meta to bite off the whole apple of "holy shit arbitrary code execution on random machines as root" is based on "everyone has their own VM, but within that VM everything is safe," so again, for it to be useful without turning into a fractal permissions nightmare, Spaces must have access to the VM contents, and at least so far appear to be intended to work as literally executing within the user's VM.
Even adding Space-scoped permissions and attributability to the socket can't really address this, this conflict between arbitrary access to inference, arbitrary access to user data, and arbitrary access to execution is really at the core of the product and that product seems to be impossible
Now there may be some meta-heads in the crowd that are like "but what about Sentinel and all the external monitoring stuff that should watch malicious botnets and blah blah blah." that's an interesting system in itself, but i plan on submitting a few more bug bounty reports in the next few days about these systems, and who knows! if meta fucking pays me for the bounty then we might never hear that part of the story.
that's all for now!
-
@jonny this is _amazing_ work but I can seriously barely believe it’s this stupid. like, intellectually I believe everything you are saying is perfectly accurate. buy emotionally even now I just can’t believe meta is this bad at engineering, this bad at product, this indifferent to harm even when the harm is directly to themselves and not externalized
-
@jonny OK, but can you make it DDOS itself?
Infinite recursion, somehow?
Agents that spawn agents, ad infinitum?
@androcat oh yes, DoS is much more trivial than that. recursive agent spawning would make it more annoying to clean up and might hook into stuff that lets it persist between container rebuilds, but to deny the system you can literally just fork bomb it as normal
-
RE: https://neuromatch.social/@jonny/117339825958098508
OK! Meta evaluated this as intended behavior, not applicable for a bug bounty, so therefore responsible disclosure no longer applies so here goes:
any process run within the VM can access the socket that provides inference with no attribution mechanism. This includes raw inference with arbitrary system and user prompts, as well as the ability to spawn agents with a toolset labeled as being for the "spaces" feature, which we will come back to.
This amounts to a horizontally contagious token and information harvesting bug being labeled as intended behavior.
Splitting details into new thread below
@jonny I’m so confused. This is wild.
-
@androcat oh yes, DoS is much more trivial than that. recursive agent spawning would make it more annoying to clean up and might hook into stuff that lets it persist between container rebuilds, but to deny the system you can literally just fork bomb it as normal
Ideally it should be done only using methods that the engineers have specifically approved
-
Now there may be some meta-heads in the crowd that are like "but what about Sentinel and all the external monitoring stuff that should watch malicious botnets and blah blah blah." that's an interesting system in itself, but i plan on submitting a few more bug bounty reports in the next few days about these systems, and who knows! if meta fucking pays me for the bounty then we might never hear that part of the story.
that's all for now!
oh! and since i dont' want to start another thread rn, meta's advertising skill just dropped! such fun! multisided market collapsing in the face of a different, much shittier multisided market: https://github.com/sneakers-the-rat/muse-skills/commit/8f8bccc3afc3e8974e7a6048940bdf4a16052690#diff-3d82b1080dcdc5db97ea500aaa83db5208deb0929d4f20342e0fbc4cfcf70359
-
@androcat oh yes, DoS is much more trivial than that. recursive agent spawning would make it more annoying to clean up and might hook into stuff that lets it persist between container rebuilds, but to deny the system you can literally just fork bomb it as normal
-
oh! and since i dont' want to start another thread rn, meta's advertising skill just dropped! such fun! multisided market collapsing in the face of a different, much shittier multisided market: https://github.com/sneakers-the-rat/muse-skills/commit/8f8bccc3afc3e8974e7a6048940bdf4a16052690#diff-3d82b1080dcdc5db97ea500aaa83db5208deb0929d4f20342e0fbc4cfcf70359
actual security researchers should totally get in on here there is a lot of stuff going on that i don't have the skills to probe that results from "what happens if you give everyone root" even from within a container. I am a fucking scrub and i keep getting my block knocked off by this thing, so i imagine someone with real skills will have a lot more fun.
