OK!
-
@jonny The Location permission combined with the right low-level permission requests provides access to a lot of information on the nearby Wi-Fi networks. Without the location permission, there's only info on the signal strength of the best available currently connected cellular and WI-Fi networks.
It would definitely be possible for us to change this by offering having spoofed values. However, it would make no sense to work on this when far more important privacy issues exist.
@GrapheneOS Or you could argue that these are fundamentals. You can choose not to use apps, browsers, etc. You can’t choose not to use the battery or network.
-
The main unprivileged body of a Space is not supposed to access the filesystem. This is enforced by.... regex
@jonny the takeaway of your thread seems to be that regex can truly solve *any* problem. The duct tape of programming
️ -
RE: https://neuromatch.social/@jonny/117339825958098508
OK! Meta evaluated this as intended behavior, not applicable for a bug bounty, so therefore responsible disclosure no longer applies so here goes:
any process run within the VM can access the socket that provides inference with no attribution mechanism. This includes raw inference with arbitrary system and user prompts, as well as the ability to spawn agents with a toolset labeled as being for the "spaces" feature, which we will come back to.
This amounts to a horizontally contagious token and information harvesting bug being labeled as intended behavior.
Splitting details into new thread below
@jonny This whole thread is so life-affirming.
️ -
@fancysandwiches i am working on this but cannot publicly disclose any progress until meta decides whether or not it is payable as a bug bounty
@jonny @fancysandwiches and, how many of the other quirks that did hit the news cycle are already from this being done lol
-
The main unprivileged body of a Space is not supposed to access the filesystem. This is enforced by.... regex
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
-
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
@jonny is that your interface? did you make a harness for this thing or is this muses web interface?
-
@jonny is that your interface? did you make a harness for this thing or is this muses web interface?
@jonny beeteedubs i linked to you a bunch in my reddit ama earlier today, so you may get a buuuunch of new followers, heh
-
@jonny is that your interface? did you make a harness for this thing or is this muses web interface?
@Viss yeah, i am comparatively spoiled by claude code and having yelled at it enough so it always writes code in a TDD loop, muse is absolutely awful at programming out of the box, but after yelling at it for several hours it was capable of generating a browser for the contents of the .jsonl files that have its raw message output. this part of muse could actually be awesome if it wasn't such a fucking trainwreck.
-
@jonny beeteedubs i linked to you a bunch in my reddit ama earlier today, so you may get a buuuunch of new followers, heh
@Viss where's the AMA at?
-
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
"So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it."
- which we've known about #LLMs since the ice age and has not changed despite their rapid advance towards super intelligence.
I think I'll forgo Meta's free VPS with #LLM carnage and stick to my Euro 2.7/month Infomaniak Swiss VPS.
-
@Viss where's the AMA at?
-
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
Hell, seeing this, if I was alibaba or tencent, I would put up poisoned packages and use the unlimited unattributable inference socket to distill meta's models, but what do I know about corporate espionage.
-
Hell, seeing this, if I was alibaba or tencent, I would put up poisoned packages and use the unlimited unattributable inference socket to distill meta's models, but what do I know about corporate espionage.
@jonny On the record? Nothing. :3
-
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
@jonny Ordering illicict Python from Alibaba
-
Hell, seeing this, if I was alibaba or tencent, I would put up poisoned packages and use the unlimited unattributable inference socket to distill meta's models, but what do I know about corporate espionage.
@jonny you think meta is that good
-
@jonny every time I come back to this thread I feel like I am having a fever dream
-
@jonny I can't believe they call these "spaces" (meta spaces?) and not "verses"
-
@jonny you think meta is that good
@ChickenPwny
No, but it would be free! -
@ChickenPwny
No, but it would be free!@jonny *should he he because AI should be open-sourced
-
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
@jonny oh great, another way that this steaming pile of shit will put even more strain on the resources of everyone hosting anything on the internet.
