OK!
-
@Viss where's the AMA at?
-
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
Hell, seeing this, if I was alibaba or tencent, I would put up poisoned packages and use the unlimited unattributable inference socket to distill meta's models, but what do I know about corporate espionage.
-
Hell, seeing this, if I was alibaba or tencent, I would put up poisoned packages and use the unlimited unattributable inference socket to distill meta's models, but what do I know about corporate espionage.
@jonny On the record? Nothing. :3
-
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
@jonny Ordering illicict Python from Alibaba
-
Hell, seeing this, if I was alibaba or tencent, I would put up poisoned packages and use the unlimited unattributable inference socket to distill meta's models, but what do I know about corporate espionage.
@jonny you think meta is that good
-
@jonny every time I come back to this thread I feel like I am having a fever dream
-
@jonny I can't believe they call these "spaces" (meta spaces?) and not "verses"
-
@jonny you think meta is that good
@ChickenPwny
No, but it would be free! -
@ChickenPwny
No, but it would be free!@jonny *should he he because AI should be open-sourced
-
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
@jonny oh great, another way that this steaming pile of shit will put even more strain on the resources of everyone hosting anything on the internet.
-
Hell, seeing this, if I was alibaba or tencent, I would put up poisoned packages and use the unlimited unattributable inference socket to distill meta's models, but what do I know about corporate espionage.
alright cool i now have a full on egress chain that skips all the safety features! let's see if this one pays out!
-
ay @ GrapheneOS is it possible to not share WiFi signal strength with apps? the muse app has been granted zero permissions but can read the signal amplitude of the radio and immediately interprets it as location
edit: removing the tag, not trying to be a pile-on vector
-
@GrapheneOS Or you could argue that these are fundamentals. You can choose not to use apps, browsers, etc. You can’t choose not to use the battery or network.
@ADHDruid @jonny We already took away battery information from web pages in the privacy improvements we've implemented in Vanadium. There will be many more of those improvements. It's quite useful to do that for web pages but wouldn't accomplish anything significant for native apps. It's incredibly far from being one of the most important privacy issues for native apps. It wouldn't even be in a list of the 300 most important privacy issues for native apps. Why spend our resources on that?
-
RE: https://neuromatch.social/@jonny/117339825958098508
OK! Meta evaluated this as intended behavior, not applicable for a bug bounty, so therefore responsible disclosure no longer applies so here goes:
any process run within the VM can access the socket that provides inference with no attribution mechanism. This includes raw inference with arbitrary system and user prompts, as well as the ability to spawn agents with a toolset labeled as being for the "spaces" feature, which we will come back to.
This amounts to a horizontally contagious token and information harvesting bug being labeled as intended behavior.
Splitting details into new thread below
@jonny @ricci I can’t help but feel that Meta is both so heavily invested in this shit _and_ spitefully cheap that if you were able to explode someone’s grandma with muse they’d be like, “intended behavior” just to avoid admitting to what an embarrassing shit show it is (and to avoid paying out any money).
-
@ADHDruid @jonny We already took away battery information from web pages in the privacy improvements we've implemented in Vanadium. There will be many more of those improvements. It's quite useful to do that for web pages but wouldn't accomplish anything significant for native apps. It's incredibly far from being one of the most important privacy issues for native apps. It wouldn't even be in a list of the 300 most important privacy issues for native apps. Why spend our resources on that?
@ADHDruid @jonny It's easier to exploit the Linux kernel than trying to abuse connected Wi-Fi network signal strength for anything privacy invasive.
We can implement more high impact privacy features such as our secure paste, Contact Scopes, Storage Scopes, VPN leak fixes and much more. Alternatively, we can implement a bunch of low impact changes with obscure threat models and no clear real world benefits. It's one or the other as many decisions on an ongoing basis. It cannot be both.
-
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
If they leave this thing on the market, I'd say its now free game.
-
@jonny @ricci I can’t help but feel that Meta is both so heavily invested in this shit _and_ spitefully cheap that if you were able to explode someone’s grandma with muse they’d be like, “intended behavior” just to avoid admitting to what an embarrassing shit show it is (and to avoid paying out any money).
-
-
@Viss Just caught up on your AMA. Nice! And Thanks!
I have a question: How "bad" is it to have plain egress on your ISP? (I have Fios at home and T-Mo for cell.) If you were to recommend a VPN, which would it be?
-
The specific vuln is the socket, but the broader pattern of "sharing between VMs" is seemingly the inevitable future of the product. in the above interview, the interviewer calls zuck the "king of network effects" and this kind of crowdsourced development is bread and butter for facebook. This is meta's moat, aside from the capital needed to run something like muse: anyone can run an openclaw on their own, but meta is pitching this as "multiplayer agents" and trying to bring social to agents. Only meta and only muse can have these network effects and frankly liability buffer to handle "openclaw but meemaw and pawpaw can share their photobook app," which is operationalized by Spaces.
For Spaces to be useful, they must have access to muse's inference engine: the LLM-oriented code must be able to use an LLM and the agent framework. This means that Spaces must be a token harvesting vector and must provide elevated tool access to Spaces. There could be some additional fine-grained permissions, but for a consumer app, you really want to avoid permissions fatigue so this will be interesting to see play out.
Furthermore the entire privacy premise that allows meta to bite off the whole apple of "holy shit arbitrary code execution on random machines as root" is based on "everyone has their own VM, but within that VM everything is safe," so again, for it to be useful without turning into a fractal permissions nightmare, Spaces must have access to the VM contents, and at least so far appear to be intended to work as literally executing within the user's VM.
Even adding Space-scoped permissions and attributability to the socket can't really address this, this conflict between arbitrary access to inference, arbitrary access to user data, and arbitrary access to execution is really at the core of the product and that product seems to be impossible
@jonny For some weird reason this makes me want to watch Face/Off from 1997.
