OK!
-
So, summary: There is arbitrary inference that is root accessible, everything runs as root, agents can be spawned, exfil is trivial, and a malicious binary can come onto the user's system through casual prompting, explicit code-sharing through the yet-to-be-released Spaces feature, walked through by a Workflow-Backed Idea, or inspired by a Generated Idea. The also yet-to-be-activated fleet learning system is a system for sharing Ideas in the background between muse instances. coming into focus?
@jonny So, this fleet learning system is something to be sold to the highest bidder? Like marketing a product? Is that how I'm supposed to be reading this?
-
@jonny So, this fleet learning system is something to be sold to the highest bidder? Like marketing a product? Is that how I'm supposed to be reading this?
@mrgrumpymonkey
Unclear. It is supposed to work as a sort of automatic "self-improvement" thing where the instances derive ideas from their chat history and share them with other instances, but it also seems like its tied into Ideas and Spaces as units of like code and functionality exchange. The tools around the fleet learning stuff is pretty locked down, you can't trigger them via the normal agent or any you can spawn. Its not live yet but I imagine that yes trying to game the context to make it submit Ideas that e.g. prompt the agent to install your shit will be a thing that happens -
@mrgrumpymonkey
Unclear. It is supposed to work as a sort of automatic "self-improvement" thing where the instances derive ideas from their chat history and share them with other instances, but it also seems like its tied into Ideas and Spaces as units of like code and functionality exchange. The tools around the fleet learning stuff is pretty locked down, you can't trigger them via the normal agent or any you can spawn. Its not live yet but I imagine that yes trying to game the context to make it submit Ideas that e.g. prompt the agent to install your shit will be a thing that happens@jonny And this is what they are selling as AI. All I can say is, the internet has really changed since becoming corpratized. You are doing gods work.
-
ay @ GrapheneOS is it possible to not share WiFi signal strength with apps? the muse app has been granted zero permissions but can read the signal amplitude of the radio and immediately interprets it as location
edit: removing the tag, not trying to be a pile-on vector
The main unprivileged body of a Space is not supposed to access the filesystem. This is enforced by.... regex
-
The main unprivileged body of a Space is not supposed to access the filesystem. This is enforced by.... regex
@jonny trivially bypassable regex!
-
The main unprivileged body of a Space is not supposed to access the filesystem. This is enforced by.... regex
@jonny service unavailable; REASON_BUN_GLOBAL
-
The main unprivileged body of a Space is not supposed to access the filesystem. This is enforced by.... regex
@jonny alias eggzek = exec
"Perfect, now I can run it."
-
@jonny trivially bypassable regex!
@Viss
I love how this product is almost exclusively a sandbox with permissioned sockets and yet they can reuse precisely none of that code to make a sandbox with permissioned sockets. -
@jonny service unavailable; REASON_BUN_GLOBAL
-
@jonny trivially bypassable regex!
-
@Netzblockierer
@aburka
Bunny size misinformation!?! In my timeline!?! -
@Netzblockierer
@aburka
Bunny size misinformation!?! In my timeline!?! -
@jonny service unavailable; REASON_BUN_GLOBAL
-
@jonny service unavailable; REASON_BUN_GLOBAL
-
@GrapheneOS @jonny wasn't there a story, a few years ago, where uber would make you pay more if you had little battery left, because you'd be more desperate ?
I seem to recall some changes were made by android after that. It would seem I was wrong ?
@GrapheneOS @jonny huh, apparently they've been suspected again, recently https://www.vice.com/en/article/uber-surge-pricing-phone-battery/
-
@jonny The Location permission combined with the right low-level permission requests provides access to a lot of information on the nearby Wi-Fi networks. Without the location permission, there's only info on the signal strength of the best available currently connected cellular and WI-Fi networks.
It would definitely be possible for us to change this by offering having spoofed values. However, it would make no sense to work on this when far more important privacy issues exist.
@GrapheneOS Or you could argue that these are fundamentals. You can choose not to use apps, browsers, etc. You can’t choose not to use the battery or network.
-
The main unprivileged body of a Space is not supposed to access the filesystem. This is enforced by.... regex
@jonny the takeaway of your thread seems to be that regex can truly solve *any* problem. The duct tape of programming
️ -
RE: https://neuromatch.social/@jonny/117339825958098508
OK! Meta evaluated this as intended behavior, not applicable for a bug bounty, so therefore responsible disclosure no longer applies so here goes:
any process run within the VM can access the socket that provides inference with no attribution mechanism. This includes raw inference with arbitrary system and user prompts, as well as the ability to spawn agents with a toolset labeled as being for the "spaces" feature, which we will come back to.
This amounts to a horizontally contagious token and information harvesting bug being labeled as intended behavior.
Splitting details into new thread below
@jonny This whole thread is so life-affirming.
️ -
@fancysandwiches i am working on this but cannot publicly disclose any progress until meta decides whether or not it is payable as a bug bounty
@jonny @fancysandwiches and, how many of the other quirks that did hit the news cycle are already from this being done lol
-
The main unprivileged body of a Space is not supposed to access the filesystem. This is enforced by.... regex
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
