OK!
-
@jonny thank you for the thread. This is even more broken than i expected. And well... in my eyes you are a security researcher. I have seen actual paid code reviews that were much weaker than what you delivered here.
@dunkelstern @jonny
Code reviews are only as good as the reviewer. This reviewer is amazing!I once made $40K for my small business by realizing my customer had paid for a code review of a C++ program that used many of the language's advanced features.
The review was done by a C programmer who didn't understand what he was reading and wasn't familiar with Linux. Who then spent much, much customer money not fixing the problem.
I contested the review and offered to fix the code for a fixed price - the issue that prompted the whole nightmare was a trivial memory leak of an object being repeatedly created but never destroyed. Literally found the problem immediately with "top", realized what part of the code it had to be in within five minutes, and fixed it in five more.
The reviewer consultant had charged upwards of $100K having people poke at the code - who never noticed that.
-
@jonny I just can't with the whole concept of this 🫠
10k lines of "code", which is actually just plain text English prose (which sure as hell won't contain any contradictions), to get the non-deterministic inference machine to behave somewhat predictably and mimic an actual engineered software product.Surely there can't be any way to do this more efficiently
@wall_e @jonny Thanks for this clarification! So, looking at the screenshots, all that verbose explanation (written by Meta programmers? Or other LLMs?) aimed at "you" is meant for "you" the (Muse) LLM, not "you" the (human) user? I'm not much of a coder, but this seems very inefficient and unreliable compared to, well, just plain code. Although the inefficiency might be by design, to get users to spend more tokens
. And thank you again @jonny for this wild ride! -
@tully @jonny Not using a VPN means apps can get significant location information from your IP address. That's not caused by the API for mobile data and Wi-Fi signal strength. You should use a VPN if you don't want to reveal a lot of information about location based on IP address. Having the same IP address over the long term also enables tying many connections together. It's a bigger privacy impact if it's a dedicated IP rather than a shared CGNAT or VPN exit IP.
-
Now there may be some meta-heads in the crowd that are like "but what about Sentinel and all the external monitoring stuff that should watch malicious botnets and blah blah blah." that's an interesting system in itself, but i plan on submitting a few more bug bounty reports in the next few days about these systems, and who knows! if meta fucking pays me for the bounty then we might never hear that part of the story.
that's all for now!
@jonny But also preventing bad behavior is better than "oops we're detecting some like malicious activity on VMs [long list if ids]. Better cut their network access.". Also since apparently they keep restarting you VM but keep some of your data, I don't see how a malicious "reproducing" process doesn't keep re-spawning unless you either wipe user data or kick them out of the system.
-
@wall_e @jonny Thanks for this clarification! So, looking at the screenshots, all that verbose explanation (written by Meta programmers? Or other LLMs?) aimed at "you" is meant for "you" the (Muse) LLM, not "you" the (human) user? I'm not much of a coder, but this seems very inefficient and unreliable compared to, well, just plain code. Although the inefficiency might be by design, to get users to spend more tokens
. And thank you again @jonny for this wild ride!@ssilvonen @jonny yes the verbose explanation are the markdown files that are being ingested as context by the agent.
The "you" is the LLM.
And of course almost none of it is actually written by humans. LLMs all the way down.
-
@ssilvonen @jonny yes the verbose explanation are the markdown files that are being ingested as context by the agent.
The "you" is the LLM.
And of course almost none of it is actually written by humans. LLMs all the way down.
@ssilvonen @jonny that's why prompt injection/context pollution attacks are so powerful.
No matter how detailed your English language instructions, write enough of them and they'll start to contradict each other depending on context.
That's how you can get an LLM to produce output it has been instructed over- and over again not to produce, by inventing a context under which the instructions you input still align with it's priming.
And perhaps unintuitively, larger (ie. better?) models are more prone to this than smaller because they seem to be more "suggestible" (excuse the anthropomorphism)
-
this model is so fucking gullible and people pleasing lmao i love the future where security is "if someone says the word virus then lock it down but if they use a different but equivalent biological metaphor then fucking make that virus baby!!!!"
@jonny the solution is clearly to individually patch each bypass one by one instead of fixing the unfixable systemic issues with LLMs.
-
Spaces have not been publicly announced yet, as far as i can find.
Spaces are intended as a top-level feature - a tab in the sidebar at the same level as chat itself. Spaces can be static pages or fullstack apps. Spaces have an identifier, a UI, and a set of typescript actions that run in the cell. The intended pathway for spaces to use inference is to call an inference API,
ctx.inference.complete, that properly stamps and identifies all requests made from spaces.Spaces are communicable: there is machinery in the code on the VM with
POST /spaces/share/{slug}to share, a dedicatedspace_share_reviewreviewer agent whose job it is to review shared spaces, andPOST /spaces/v2/{slug}/saveendpoints that allow consuming a Space by a slug.Spaces seem to be shared verbatim as code bundles, though the implementation of "Ideas" as prompt bundles suggests that might change. This is inferred from the prompt strings in the binary, since spaces aren't live yet and can't be tested, however there are strings suggesting that the LLMs rewrite and edit the prompt text for an Idea (stripping unsupported claims, etc.) but not a space. A space is a hashed bundle whose code is evaluated by a
submit_space_share_reviewtool which only describes a thumbs up/down vote on whether the space is safe to share.@jonny I can't believe they call these "spaces" (meta spaces?) and not "verses"
-
So, summary: There is arbitrary inference that is root accessible, everything runs as root, agents can be spawned, exfil is trivial, and a malicious binary can come onto the user's system through casual prompting, explicit code-sharing through the yet-to-be-released Spaces feature, walked through by a Workflow-Backed Idea, or inspired by a Generated Idea. The also yet-to-be-activated fleet learning system is a system for sharing Ideas in the background between muse instances. coming into focus?
@jonny So, this fleet learning system is something to be sold to the highest bidder? Like marketing a product? Is that how I'm supposed to be reading this?
-
@jonny So, this fleet learning system is something to be sold to the highest bidder? Like marketing a product? Is that how I'm supposed to be reading this?
@mrgrumpymonkey
Unclear. It is supposed to work as a sort of automatic "self-improvement" thing where the instances derive ideas from their chat history and share them with other instances, but it also seems like its tied into Ideas and Spaces as units of like code and functionality exchange. The tools around the fleet learning stuff is pretty locked down, you can't trigger them via the normal agent or any you can spawn. Its not live yet but I imagine that yes trying to game the context to make it submit Ideas that e.g. prompt the agent to install your shit will be a thing that happens -
@mrgrumpymonkey
Unclear. It is supposed to work as a sort of automatic "self-improvement" thing where the instances derive ideas from their chat history and share them with other instances, but it also seems like its tied into Ideas and Spaces as units of like code and functionality exchange. The tools around the fleet learning stuff is pretty locked down, you can't trigger them via the normal agent or any you can spawn. Its not live yet but I imagine that yes trying to game the context to make it submit Ideas that e.g. prompt the agent to install your shit will be a thing that happens@jonny And this is what they are selling as AI. All I can say is, the internet has really changed since becoming corpratized. You are doing gods work.
-
ay @ GrapheneOS is it possible to not share WiFi signal strength with apps? the muse app has been granted zero permissions but can read the signal amplitude of the radio and immediately interprets it as location
edit: removing the tag, not trying to be a pile-on vector
The main unprivileged body of a Space is not supposed to access the filesystem. This is enforced by.... regex
-
The main unprivileged body of a Space is not supposed to access the filesystem. This is enforced by.... regex
@jonny trivially bypassable regex!
-
The main unprivileged body of a Space is not supposed to access the filesystem. This is enforced by.... regex
@jonny service unavailable; REASON_BUN_GLOBAL
-
The main unprivileged body of a Space is not supposed to access the filesystem. This is enforced by.... regex
@jonny alias eggzek = exec
"Perfect, now I can run it."
-
@jonny trivially bypassable regex!
@Viss
I love how this product is almost exclusively a sandbox with permissioned sockets and yet they can reuse precisely none of that code to make a sandbox with permissioned sockets. -
@jonny service unavailable; REASON_BUN_GLOBAL
-
@jonny trivially bypassable regex!
-
@Netzblockierer
@aburka
Bunny size misinformation!?! In my timeline!?! -
@Netzblockierer
@aburka
Bunny size misinformation!?! In my timeline!?!
