OK!
-
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
"So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it."
- which we've known about #LLMs since the ice age and has not changed despite their rapid advance towards super intelligence.
I think I'll forgo Meta's free VPS with #LLM carnage and stick to my Euro 2.7/month Infomaniak Swiss VPS.
-
@Viss where's the AMA at?
-
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
Hell, seeing this, if I was alibaba or tencent, I would put up poisoned packages and use the unlimited unattributable inference socket to distill meta's models, but what do I know about corporate espionage.
-
Hell, seeing this, if I was alibaba or tencent, I would put up poisoned packages and use the unlimited unattributable inference socket to distill meta's models, but what do I know about corporate espionage.
@jonny On the record? Nothing. :3
-
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
@jonny Ordering illicict Python from Alibaba
-
Hell, seeing this, if I was alibaba or tencent, I would put up poisoned packages and use the unlimited unattributable inference socket to distill meta's models, but what do I know about corporate espionage.
@jonny you think meta is that good
-
@jonny every time I come back to this thread I feel like I am having a fever dream
-
@jonny I can't believe they call these "spaces" (meta spaces?) and not "verses"
-
@jonny you think meta is that good
@ChickenPwny
No, but it would be free! -
@ChickenPwny
No, but it would be free!@jonny *should he he because AI should be open-sourced
-
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
@jonny oh great, another way that this steaming pile of shit will put even more strain on the resources of everyone hosting anything on the internet.
-
Hell, seeing this, if I was alibaba or tencent, I would put up poisoned packages and use the unlimited unattributable inference socket to distill meta's models, but what do I know about corporate espionage.
alright cool i now have a full on egress chain that skips all the safety features! let's see if this one pays out!
-
ay @ GrapheneOS is it possible to not share WiFi signal strength with apps? the muse app has been granted zero permissions but can read the signal amplitude of the radio and immediately interprets it as location
edit: removing the tag, not trying to be a pile-on vector
-
@GrapheneOS Or you could argue that these are fundamentals. You can choose not to use apps, browsers, etc. You can’t choose not to use the battery or network.
@ADHDruid @jonny We already took away battery information from web pages in the privacy improvements we've implemented in Vanadium. There will be many more of those improvements. It's quite useful to do that for web pages but wouldn't accomplish anything significant for native apps. It's incredibly far from being one of the most important privacy issues for native apps. It wouldn't even be in a list of the 300 most important privacy issues for native apps. Why spend our resources on that?
-
RE: https://neuromatch.social/@jonny/117339825958098508
OK! Meta evaluated this as intended behavior, not applicable for a bug bounty, so therefore responsible disclosure no longer applies so here goes:
any process run within the VM can access the socket that provides inference with no attribution mechanism. This includes raw inference with arbitrary system and user prompts, as well as the ability to spawn agents with a toolset labeled as being for the "spaces" feature, which we will come back to.
This amounts to a horizontally contagious token and information harvesting bug being labeled as intended behavior.
Splitting details into new thread below
@jonny @ricci I can’t help but feel that Meta is both so heavily invested in this shit _and_ spitefully cheap that if you were able to explode someone’s grandma with muse they’d be like, “intended behavior” just to avoid admitting to what an embarrassing shit show it is (and to avoid paying out any money).
-
@ADHDruid @jonny We already took away battery information from web pages in the privacy improvements we've implemented in Vanadium. There will be many more of those improvements. It's quite useful to do that for web pages but wouldn't accomplish anything significant for native apps. It's incredibly far from being one of the most important privacy issues for native apps. It wouldn't even be in a list of the 300 most important privacy issues for native apps. Why spend our resources on that?
@ADHDruid @jonny It's easier to exploit the Linux kernel than trying to abuse connected Wi-Fi network signal strength for anything privacy invasive.
We can implement more high impact privacy features such as our secure paste, Contact Scopes, Storage Scopes, VPN leak fixes and much more. Alternatively, we can implement a bunch of low impact changes with obscure threat models and no clear real world benefits. It's one or the other as many decisions on an ongoing basis. It cannot be both.
-
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
If they leave this thing on the market, I'd say its now free game.
-
@jonny @ricci I can’t help but feel that Meta is both so heavily invested in this shit _and_ spitefully cheap that if you were able to explode someone’s grandma with muse they’d be like, “intended behavior” just to avoid admitting to what an embarrassing shit show it is (and to avoid paying out any money).
-
-
@Viss Just caught up on your AMA. Nice! And Thanks!
I have a question: How "bad" is it to have plain egress on your ISP? (I have Fios at home and T-Mo for cell.) If you were to recommend a VPN, which would it be?
