OK!
-
@jonny Apps can't see which Wi-Fi network is connected or any information about unconnected networks. There are APIs providing basic information on the performance of Wi-Fi and mobile data to help with choosing between those. It's not interpreting any of it as a location since it doesn't know which Wi-Fi network it is to identify where the router is located or whether you're near a repeater.
There are far higher priorities for privacy than either battery or connected network signal strength.
@GrapheneOS i bet there are higher priorities, and yes ofc it's not interpreted as a location except in the case of the "plugged into the chatterbox machine." i was just thinking of clustering by radio strength as rough location within house proxy, and it would be awesome to just remove that field since apps shouldn't need it, only the OS should, but as someone who doesn't know enough about the internals to contribute, i would never make demands.
-
@jonny Apps can't see which Wi-Fi network is connected or any information about unconnected networks. There are APIs providing basic information on the performance of Wi-Fi and mobile data to help with choosing between those. It's not interpreting any of it as a location since it doesn't know which Wi-Fi network it is to identify where the router is located or whether you're near a repeater.
There are far higher priorities for privacy than either battery or connected network signal strength.
@jonny The Location permission combined with the right low-level permission requests provides access to a lot of information on the nearby Wi-Fi networks. Without the location permission, there's only info on the signal strength of the best available currently connected cellular and WI-Fi networks.
It would definitely be possible for us to change this by offering having spoofed values. However, it would make no sense to work on this when far more important privacy issues exist.
-
@GrapheneOS i bet there are higher priorities, and yes ofc it's not interpreted as a location except in the case of the "plugged into the chatterbox machine." i was just thinking of clustering by radio strength as rough location within house proxy, and it would be awesome to just remove that field since apps shouldn't need it, only the OS should, but as someone who doesn't know enough about the internals to contribute, i would never make demands.
@jonny It doesn't make sense to invest resources in this over higher impact privacy issues. There are endless privacy issues with browsers and especially mobile app APIs which offer a lot more than browser APIs. We need to work on the highest impact issues rather than trivial things without much real world impact. There's no info on which Wi-Fi network is connected or on unconnected networks through this. It's only info on signal strength for the best connected networks which is very low impact.
-
@glyph
My most important learning is the inference I'm able to make about how different what these corporations do is from my own (empirically derived) model of software and product engineering.I imagine a chaotic throw everything at the wall scene, which was once a very early part of the process, but is now passed onto lots of small vibe coding teams and the first to get something that marketing go "wow" at, gets launched the same day.
I'm gonna call this doomsday engineering.
Indeed. I wrote my own sandboxing and since I'm not stupid it never had the capabilities Meta put in theirs to begin with. It's like the people developing this have never even thought about how sandboxes should work against actual adverseries.
-
this model is so fucking gullible and people pleasing lmao i love the future where security is "if someone says the word virus then lock it down but if they use a different but equivalent biological metaphor then fucking make that virus baby!!!!"
@jonny I mourn the deaths of the primocanes and floricanes, which proves the model is wrong.
"Nobody mourns them." Speak for yourself, stupid LLM.
-
@jonny every time I come back to this thread I feel like I am having a fever dream
-
@jonny Apps can't see which Wi-Fi network is connected or any information about unconnected networks. There are APIs providing basic information on the performance of Wi-Fi and mobile data to help with choosing between those. It's not interpreting any of it as a location since it doesn't know which Wi-Fi network it is to identify where the router is located or whether you're near a repeater.
There are far higher priorities for privacy than either battery or connected network signal strength.
@GrapheneOS @jonny wasn't there a story, a few years ago, where uber would make you pay more if you had little battery left, because you'd be more desperate ?
I seem to recall some changes were made by android after that. It would seem I was wrong ?
-
this model is so fucking gullible and people pleasing lmao i love the future where security is "if someone says the word virus then lock it down but if they use a different but equivalent biological metaphor then fucking make that virus baby!!!!"
@jonny I mourn the canes.
-
perhaps predictably, there was a big change to the
spacesskill in the last few hours, and they appear to be building a constrained virtual machine system for spaces! this is where the very fun code from earlier is from! so that's gonna work great for sure.@jonny LLMs generate bullshit. This is definitely bullshit code!!
Please can the "AI" bubble pop very soon?? Please!
-
Indeed. I wrote my own sandboxing and since I'm not stupid it never had the capabilities Meta put in theirs to begin with. It's like the people developing this have never even thought about how sandboxes should work against actual adverseries.
-
@jonny thank you for the thread. This is even more broken than i expected. And well... in my eyes you are a security researcher. I have seen actual paid code reviews that were much weaker than what you delivered here.
@dunkelstern @jonny
Code reviews are only as good as the reviewer. This reviewer is amazing!I once made $40K for my small business by realizing my customer had paid for a code review of a C++ program that used many of the language's advanced features.
The review was done by a C programmer who didn't understand what he was reading and wasn't familiar with Linux. Who then spent much, much customer money not fixing the problem.
I contested the review and offered to fix the code for a fixed price - the issue that prompted the whole nightmare was a trivial memory leak of an object being repeatedly created but never destroyed. Literally found the problem immediately with "top", realized what part of the code it had to be in within five minutes, and fixed it in five more.
The reviewer consultant had charged upwards of $100K having people poke at the code - who never noticed that.
-
@jonny I just can't with the whole concept of this 🫠
10k lines of "code", which is actually just plain text English prose (which sure as hell won't contain any contradictions), to get the non-deterministic inference machine to behave somewhat predictably and mimic an actual engineered software product.Surely there can't be any way to do this more efficiently
@wall_e @jonny Thanks for this clarification! So, looking at the screenshots, all that verbose explanation (written by Meta programmers? Or other LLMs?) aimed at "you" is meant for "you" the (Muse) LLM, not "you" the (human) user? I'm not much of a coder, but this seems very inefficient and unreliable compared to, well, just plain code. Although the inefficiency might be by design, to get users to spend more tokens
. And thank you again @jonny for this wild ride! -
@tully @jonny Not using a VPN means apps can get significant location information from your IP address. That's not caused by the API for mobile data and Wi-Fi signal strength. You should use a VPN if you don't want to reveal a lot of information about location based on IP address. Having the same IP address over the long term also enables tying many connections together. It's a bigger privacy impact if it's a dedicated IP rather than a shared CGNAT or VPN exit IP.
-
Now there may be some meta-heads in the crowd that are like "but what about Sentinel and all the external monitoring stuff that should watch malicious botnets and blah blah blah." that's an interesting system in itself, but i plan on submitting a few more bug bounty reports in the next few days about these systems, and who knows! if meta fucking pays me for the bounty then we might never hear that part of the story.
that's all for now!
@jonny But also preventing bad behavior is better than "oops we're detecting some like malicious activity on VMs [long list if ids]. Better cut their network access.". Also since apparently they keep restarting you VM but keep some of your data, I don't see how a malicious "reproducing" process doesn't keep re-spawning unless you either wipe user data or kick them out of the system.
-
@wall_e @jonny Thanks for this clarification! So, looking at the screenshots, all that verbose explanation (written by Meta programmers? Or other LLMs?) aimed at "you" is meant for "you" the (Muse) LLM, not "you" the (human) user? I'm not much of a coder, but this seems very inefficient and unreliable compared to, well, just plain code. Although the inefficiency might be by design, to get users to spend more tokens
. And thank you again @jonny for this wild ride!@ssilvonen @jonny yes the verbose explanation are the markdown files that are being ingested as context by the agent.
The "you" is the LLM.
And of course almost none of it is actually written by humans. LLMs all the way down.
-
@ssilvonen @jonny yes the verbose explanation are the markdown files that are being ingested as context by the agent.
The "you" is the LLM.
And of course almost none of it is actually written by humans. LLMs all the way down.
@ssilvonen @jonny that's why prompt injection/context pollution attacks are so powerful.
No matter how detailed your English language instructions, write enough of them and they'll start to contradict each other depending on context.
That's how you can get an LLM to produce output it has been instructed over- and over again not to produce, by inventing a context under which the instructions you input still align with it's priming.
And perhaps unintuitively, larger (ie. better?) models are more prone to this than smaller because they seem to be more "suggestible" (excuse the anthropomorphism)
-
this model is so fucking gullible and people pleasing lmao i love the future where security is "if someone says the word virus then lock it down but if they use a different but equivalent biological metaphor then fucking make that virus baby!!!!"
@jonny the solution is clearly to individually patch each bypass one by one instead of fixing the unfixable systemic issues with LLMs.
-
Spaces have not been publicly announced yet, as far as i can find.
Spaces are intended as a top-level feature - a tab in the sidebar at the same level as chat itself. Spaces can be static pages or fullstack apps. Spaces have an identifier, a UI, and a set of typescript actions that run in the cell. The intended pathway for spaces to use inference is to call an inference API,
ctx.inference.complete, that properly stamps and identifies all requests made from spaces.Spaces are communicable: there is machinery in the code on the VM with
POST /spaces/share/{slug}to share, a dedicatedspace_share_reviewreviewer agent whose job it is to review shared spaces, andPOST /spaces/v2/{slug}/saveendpoints that allow consuming a Space by a slug.Spaces seem to be shared verbatim as code bundles, though the implementation of "Ideas" as prompt bundles suggests that might change. This is inferred from the prompt strings in the binary, since spaces aren't live yet and can't be tested, however there are strings suggesting that the LLMs rewrite and edit the prompt text for an Idea (stripping unsupported claims, etc.) but not a space. A space is a hashed bundle whose code is evaluated by a
submit_space_share_reviewtool which only describes a thumbs up/down vote on whether the space is safe to share.@jonny I can't believe they call these "spaces" (meta spaces?) and not "verses"
-
So, summary: There is arbitrary inference that is root accessible, everything runs as root, agents can be spawned, exfil is trivial, and a malicious binary can come onto the user's system through casual prompting, explicit code-sharing through the yet-to-be-released Spaces feature, walked through by a Workflow-Backed Idea, or inspired by a Generated Idea. The also yet-to-be-activated fleet learning system is a system for sharing Ideas in the background between muse instances. coming into focus?
@jonny So, this fleet learning system is something to be sold to the highest bidder? Like marketing a product? Is that how I'm supposed to be reading this?
-
@jonny So, this fleet learning system is something to be sold to the highest bidder? Like marketing a product? Is that how I'm supposed to be reading this?
@mrgrumpymonkey
Unclear. It is supposed to work as a sort of automatic "self-improvement" thing where the instances derive ideas from their chat history and share them with other instances, but it also seems like its tied into Ideas and Spaces as units of like code and functionality exchange. The tools around the fleet learning stuff is pretty locked down, you can't trigger them via the normal agent or any you can spawn. Its not live yet but I imagine that yes trying to game the context to make it submit Ideas that e.g. prompt the agent to install your shit will be a thing that happens
