OK!
-
RE: https://neuromatch.social/@jonny/117339825958098508
OK! Meta evaluated this as intended behavior, not applicable for a bug bounty, so therefore responsible disclosure no longer applies so here goes:
any process run within the VM can access the socket that provides inference with no attribution mechanism. This includes raw inference with arbitrary system and user prompts, as well as the ability to spawn agents with a toolset labeled as being for the "spaces" feature, which we will come back to.
This amounts to a horizontally contagious token and information harvesting bug being labeled as intended behavior.
Splitting details into new thread below
@jonny @ricci I can’t help but feel that Meta is both so heavily invested in this shit _and_ spitefully cheap that if you were able to explode someone’s grandma with muse they’d be like, “intended behavior” just to avoid admitting to what an embarrassing shit show it is (and to avoid paying out any money).
-
@ADHDruid @jonny We already took away battery information from web pages in the privacy improvements we've implemented in Vanadium. There will be many more of those improvements. It's quite useful to do that for web pages but wouldn't accomplish anything significant for native apps. It's incredibly far from being one of the most important privacy issues for native apps. It wouldn't even be in a list of the 300 most important privacy issues for native apps. Why spend our resources on that?
@ADHDruid @jonny It's easier to exploit the Linux kernel than trying to abuse connected Wi-Fi network signal strength for anything privacy invasive.
We can implement more high impact privacy features such as our secure paste, Contact Scopes, Storage Scopes, VPN leak fixes and much more. Alternatively, we can implement a bunch of low impact changes with obscure threat models and no clear real world benefits. It's one or the other as many decisions on an ongoing basis. It cannot be both.
-
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
If they leave this thing on the market, I'd say its now free game.
-
@jonny @ricci I can’t help but feel that Meta is both so heavily invested in this shit _and_ spitefully cheap that if you were able to explode someone’s grandma with muse they’d be like, “intended behavior” just to avoid admitting to what an embarrassing shit show it is (and to avoid paying out any money).
-
-
@Viss Just caught up on your AMA. Nice! And Thanks!
I have a question: How "bad" is it to have plain egress on your ISP? (I have Fios at home and T-Mo for cell.) If you were to recommend a VPN, which would it be?
-
The specific vuln is the socket, but the broader pattern of "sharing between VMs" is seemingly the inevitable future of the product. in the above interview, the interviewer calls zuck the "king of network effects" and this kind of crowdsourced development is bread and butter for facebook. This is meta's moat, aside from the capital needed to run something like muse: anyone can run an openclaw on their own, but meta is pitching this as "multiplayer agents" and trying to bring social to agents. Only meta and only muse can have these network effects and frankly liability buffer to handle "openclaw but meemaw and pawpaw can share their photobook app," which is operationalized by Spaces.
For Spaces to be useful, they must have access to muse's inference engine: the LLM-oriented code must be able to use an LLM and the agent framework. This means that Spaces must be a token harvesting vector and must provide elevated tool access to Spaces. There could be some additional fine-grained permissions, but for a consumer app, you really want to avoid permissions fatigue so this will be interesting to see play out.
Furthermore the entire privacy premise that allows meta to bite off the whole apple of "holy shit arbitrary code execution on random machines as root" is based on "everyone has their own VM, but within that VM everything is safe," so again, for it to be useful without turning into a fractal permissions nightmare, Spaces must have access to the VM contents, and at least so far appear to be intended to work as literally executing within the user's VM.
Even adding Space-scoped permissions and attributability to the socket can't really address this, this conflict between arbitrary access to inference, arbitrary access to user data, and arbitrary access to execution is really at the core of the product and that product seems to be impossible
@jonny For some weird reason this makes me want to watch Face/Off from 1997.
-
@Viss Just caught up on your AMA. Nice! And Thanks!
I have a question: How "bad" is it to have plain egress on your ISP? (I have Fios at home and T-Mo for cell.) If you were to recommend a VPN, which would it be?
@gregatron5 i tunnel everything i can get away with. id use my own
https://hax.lol/zipline
pw: lolcorpovpns -
@ADHDruid @jonny We already took away battery information from web pages in the privacy improvements we've implemented in Vanadium. There will be many more of those improvements. It's quite useful to do that for web pages but wouldn't accomplish anything significant for native apps. It's incredibly far from being one of the most important privacy issues for native apps. It wouldn't even be in a list of the 300 most important privacy issues for native apps. Why spend our resources on that?
@GrapheneOS @ADHDruid @jonny
So you do agree that there are valid privacy concerns related to providing extraneous information, such as battery level, to third parties, as that is a directly advertised feature of the Vanadium Browser on GrapheneOS (https://grapheneos.org/features#vanadium).
But that same concern isn't extended to native apps?I get resources being an issue, but at the same time whataboutism-ing peoples privacy concerns is not a good way to do PR.
Even the most basic "We've heard your concerns and have added it to our nice-to-have feature request list" would go a long way in quelling people's concerns and queries. -
@GrapheneOS @ADHDruid @jonny
So you do agree that there are valid privacy concerns related to providing extraneous information, such as battery level, to third parties, as that is a directly advertised feature of the Vanadium Browser on GrapheneOS (https://grapheneos.org/features#vanadium).
But that same concern isn't extended to native apps?I get resources being an issue, but at the same time whataboutism-ing peoples privacy concerns is not a good way to do PR.
Even the most basic "We've heard your concerns and have added it to our nice-to-have feature request list" would go a long way in quelling people's concerns and queries.@hailhydra @ADHDruid @jonny You're completely misrepresenting what we've said in our posts.
Native applications aren't web apps. They run in a drastically weaker sandbox and have access to far more information about the device. There's a massive gap between what's a serious privacy issue for a web page and a native app.
These things are nearly entirely inconsequential for native apps when considering how many far more serious privacy issues need to be addressed. We're working on those issues.
-
So again, how could one end up with a compromised package on a muse instance? Wouldn't that have to be some sophisticated supply chain attack? Nope! Muse attempted to install packages from PyPI, which caused a card to pop up on the user interface asking for me to approve connecting to PyPI. I was not watching the screen, so the request timed out. It then proceeded to raw dog a list of PyPI mirrors from its training data. It couldn't figure out how to use
uv, so it then generated a wheel download script that would bypass any lockfile that validated packages by hash. It forked that to the background, forgot about it, and then proceeded to attempt to manually download the specified dependencies across a dozen or two tool calls with direct URL construction over whatever mirrors returned something.Connecting to PyPI required explicit approval, but connecting to the mirrors didn't, and so i wouldn't have even noticed if i didn't always read the raw message stream rather than the interface output because you can never trust these things. When I stopped it and said "don't connect to random pypi mirrors wtf are you doing" it 1) lied about PyPI being unreachable because it has no visibility into the permission status and by pattern words should be there, 2) told me that two of the mirrors it tried were official PyPI mirrors, and 3) presented randomly wandering PyPI indexes as if it was a normal thing to do. If I wasn't a python developer and knew already there are no official PyPI mirrors, and also actively investigating how its egress permissions worked, I probably would have just accepted that.
So anyway, unless you are a user with lots of direct domain knowledge about a language packaging ecosystem who is reading the entire raw message log as it happens, muse will aggressively download random shit from the internet and execute it.
@jonny in the current hellish state of software, is there really a difference between pypi and its "mirrors"? both can serve you malicious code if the package author wants it.
-
@jonny in the current hellish state of software, is there really a difference between pypi and its "mirrors"? both can serve you malicious code if the package author wants it.
@astromateusz
Sure, but when meemaw says "make me a photo album" and the agent goes to install imagemagick somewhere deep in the dep tree of whatever its doing, it should at least get the real imagemagick and not "imagemagick but it has been typosquatted to be a ransomware vector" -
@astromateusz
Sure, but when meemaw says "make me a photo album" and the agent goes to install imagemagick somewhere deep in the dep tree of whatever its doing, it should at least get the real imagemagick and not "imagemagick but it has been typosquatted to be a ransomware vector"@astromateusz
The surface of what could be a virus at any point is always delicate, the social nature of open source has a kind of immune system for this, but LLMs do not. So the ability to randomly install malware because the RNG rolled "interpret timeouts as PyPI being down" that day is coproductive with more bugs and shit making otherwise good programs accidental malware. But its good to be specific about the kinds of harms that these consumer products impose even if they are interwoven with countless other more esoterically broken systems like repository governance -
@astromateusz
The surface of what could be a virus at any point is always delicate, the social nature of open source has a kind of immune system for this, but LLMs do not. So the ability to randomly install malware because the RNG rolled "interpret timeouts as PyPI being down" that day is coproductive with more bugs and shit making otherwise good programs accidental malware. But its good to be specific about the kinds of harms that these consumer products impose even if they are interwoven with countless other more esoterically broken systems like repository governance@jonny
true, tbh I'm just bitter about the current state of Python ecosystem since a lot of large packages maintainers either gave up and allow slop in their code or are fully on the hype train. -
@hailhydra @ADHDruid @jonny You're completely misrepresenting what we've said in our posts.
Native applications aren't web apps. They run in a drastically weaker sandbox and have access to far more information about the device. There's a massive gap between what's a serious privacy issue for a web page and a native app.
These things are nearly entirely inconsequential for native apps when considering how many far more serious privacy issues need to be addressed. We're working on those issues.
@GrapheneOS @ADHDruid @jonny
Yes I fully understand the different environments and threat models between a browser sandboxed web-app, and an android native app.What I'm trying to say is that the average user is just going to see you dismissing what they might feel is valid concerns (whether they are valid or not is irrelevant) and feel disgruntled towards the project.
What you're saying isn't "we don't think this is a concern" or "this doesn't fit the threat model grapheneos is designed against". What you're saying is "we don't think this is *enough* of a concern for us to fix". It would have been much better PR to just say "your concern has been acknowledged" and silently shove the feature request in the "TODO-2034" pile internally, if you do anything at all.
-
alright cool i now have a full on egress chain that skips all the safety features! let's see if this one pays out!
most people around here already correctly hate it because it's a heinous surveillance product, but even if you are big into AI, it's just a really fuckin shitty agent. I'm going to speak to a different audience for a second, so don't go misconstruing this as an endorsement of the category of technologies as it exists now, even though i think there is some plausible application for small local models as brute force interface glue. but also, since i know most ppl here are abstinent, this might read as a bit over-explainy to people who use these things regularly, so everyone just keep calm online.
it is terrible at turn and task management, codex + openai's models and claude code both handle mid-turn additions/amendments well, but if you say anything mid-turn it completely derails muse. That is completely essential for an "every day agent for the non-technically inclined" where people are expected to chat freely with it like an assistant. Like the canonical ad fantasy is the busy executive woman darting around her office going "robot! i need this, no wait robot! also that!" and that is exactly what it does worse than any other thing of its kind.
the context management is a fucking soup. The context window is the whole input to an LLM. There are a lot of extra surrounding ~ things ~ that can happen, but fundamentally, controlling what is in a context window is the task of using one, and filling the context window in different ways so that it can interact with different kinds of things well is what different app surfaces are. Scaffolding information so that it can selectively load a context that steers the output correctly is the only way it is possible to do anything more complex than the size of a single context window. (i don't really think that this is analogous to 'abstraction', in my experience thinking about it more like database indices is closer). If you just try and load everything, eventually the LLM becomes unusable because attention is just a parlor trick and at that scale it really shows - it can't attend to everything, and it can't do what humans do which is have an intrinsic sense of the meaning, interaction, setting, etc. of information, so it attends to anything and does whatever.
the idiom of projects as contexts as directories is pretty good, not perfect but ok - there is a reason that every time you start a new session with other agents, the first thing they do is run out and load their context with a hierarchy of pointers. Importantly, they do not go and read every project you have on your computer. Meta is the rich kid who bought the most expensive ferrari on the lot by giving everyone a VM but they don't have a drivers license so they just stand around it telling people how cool it looks. they have a whole fucking filesystem and they have done nothing with it, the only structure the app imposes is for the surveillance information, but the rest is just a huge free for all. The main chat is literally a continuous context window that compacts context going back all the way to when you started using the app. The last compaction literally contains abandoned roleplay quotes from when i was first trying to break down its system prompt resistance. The
MEMORY.mdthat gets loaded into every context window is a bullet point list of basically everything the agent has ever done in chronological order. I've tried to get it to not do that but it actually insists and says that's what it's for. There is no mechanism for clearing context.Having "side chats" as the only means of context structure is fuckin laughable. If you wanted to do that, you would need to have some way of passing information back and forth between them the same way that subagent spawning or being able to consume the context of another project works. Instead there is no means of sharing information between chats at all, so every chat starts out as the worst of both worlds, a total amnesiac riddled with irrelevant information from weeks ago across the semantic universe. They don't even know about the existence of other chats except for as a UI feature, and I have had to go from telling it to grep its own fucking logs to writing a database with an api for it so it has some mechanism for recalling things that were said. (just so it's clear, i am not settling into just using this thing, this is out of frustration but control of context is also an important part of adversarial use, because otherwise the thing writes in a bunch of safety rules everywhere, so i need to give it mechanisms under my control for recall and the incentive to leave things out of its context compactions by giving it a narrative alternative. context control is model control, modulo extra-inference safeguards.).
Project contexts have an obvious ux analogy as context tabs that get declared or derived during the continual self-improvement consolidation sweeps. This thing is built with the fucking markdown disease which is the most baffling feature of the LLM landscape. If these things are so fucking advanced they are escaping our comprehension, why don't they store their memory in some fuckass idiolanguistic borg gibberish binary graph, why does their entire being have to be fucking encyclopedias worth of corporate top gun one liners? But that dooms this kind of product.
Coding harnesses work because code has a unitized context. The entire universe of code that works is made of packages. It might not be neat as a honeycomb, there's lots of leakage and jank, but good code has scope, focus. boundary shit. A whole life agent must be able to nimbly juggle context that does not have clean boundaries. It is going to be taking a two story beer bong of your work email and then eat a gigabyte of recipe blogs. Peoples lives have so much shit in them that don't all have to do with one another, and the app can't be hacking into the HR system to check the next scheduled sick leave when someone asks it what time their doctors appointment is!
This problem of managing heterogeneous graphs of unrelated data was what i wrote this whole fucking book about the relationship between knowledge graphs and the cloud and AI about. I thought that the obvious form they would take is to be strapped on to graph databases because that is a natural match to the problem of being a magical interface glue you can wrap around surveillance to do mass mentalism with. I feel like we are suffering a somehow worse timeline where CERN threw our shit into the parallel universe where total fuckin bozo shit got a game breaking buff and then the dev died. Our fucking markdown apocalypse is a temu ass apocalypse.
They could have even faked it. They have these constant "self improvement" passes that are just like pointless anxiety dreams. They are burning money to reprocess everything that happens over and over for fucking nothing. Even given the lossy and probabilistic and unpredictable nature of this technology, if i was in a product role on this i would have been like "CAN WE MAKE IT ORGANIZE THE STUFF PEOPLE SAY INTO GROUPS???" The system prompts use the fake fucking wikilinks to nowhere tic but like WHAT IF THERE WERE ACTUAL LINKS AND A DATABASE TO RESOLVE THEM. The LLMs can actually do that kind of tool use, even if it's like trying to plug in a USB where sometimes it fails because they try and put a social security number into the first name hole and you need to flip it around a few times. From that kind of recurring re-processing waste they could have made a deduplicating, topically indexed memory that could be resolved dynamically, selected by a context tab in the sidebar like "car stuff" or "healthcare" or whatever that resolved in a graph query over your fucking precious markdown kingdom. It would be wrong but it would at least be more similar to what is actually needed. It is almost more frustrating to me that instead of being some fiendishly cleverly designed technological supervirus it's just the most halfassed cardboard dumbass trap and it will still have the bad effect. What it is useful for is investigating itself because it has privileged tools to do so, otherwise, if you wanted to, every other way you could run an agent would be better than this.
so i don't want to hear that i hate this app because i'm just an AI hater. because like, yeah, i am, but also i hate it in part because it sucks. I don't think "they are all shitty and can do nothing so what did you expect" is a useful critical perspective, both because it's not really true - they can indeed do things, even if I think the circle around which things is much smaller than the maximalists. Moreso it doesn't engage with the subtlety of how they fail and why, which is essential for knowing what they really can't do and making a remotely compelling case to anyone who is not abstinent on principle. Like the reason it's failing is because of the limits of what a probabilistic text generator can do when trapped in a systemd prison of markdown, and because the technology is stochastic black box as a service, there isn't really a good way of determining those limits except for empirically. I resent having to know any of this to be able to understand what is happening around me, but i'm looking at the thing for what it is and it's a busted miracle. It's cool that meta can afford to float the liability and compute costs for running a vm for every person on earth, people should be able to control computers, with you on that, but this is the monkey's paw version of that idea. So that part is a miracle. We condemned our children and grandchildren to a climate hell in one great blaze of brute force grift that managed to make a few web apps.
Meta has done it again, the way only meta can, spend the most amount of money to do the shittiest thing you have ever seen.
-
@GrapheneOS Or you could argue that these are fundamentals. You can choose not to use apps, browsers, etc. You can’t choose not to use the battery or network.
@GrapheneOS Look, I appreciate the work you do, but I don’t understand the fixation on the web browser. There’s plenty of vectors that utilise native apps and local lookup.
But what do I know, I only work in TSCM.
@jonny -
most people around here already correctly hate it because it's a heinous surveillance product, but even if you are big into AI, it's just a really fuckin shitty agent. I'm going to speak to a different audience for a second, so don't go misconstruing this as an endorsement of the category of technologies as it exists now, even though i think there is some plausible application for small local models as brute force interface glue. but also, since i know most ppl here are abstinent, this might read as a bit over-explainy to people who use these things regularly, so everyone just keep calm online.
it is terrible at turn and task management, codex + openai's models and claude code both handle mid-turn additions/amendments well, but if you say anything mid-turn it completely derails muse. That is completely essential for an "every day agent for the non-technically inclined" where people are expected to chat freely with it like an assistant. Like the canonical ad fantasy is the busy executive woman darting around her office going "robot! i need this, no wait robot! also that!" and that is exactly what it does worse than any other thing of its kind.
the context management is a fucking soup. The context window is the whole input to an LLM. There are a lot of extra surrounding ~ things ~ that can happen, but fundamentally, controlling what is in a context window is the task of using one, and filling the context window in different ways so that it can interact with different kinds of things well is what different app surfaces are. Scaffolding information so that it can selectively load a context that steers the output correctly is the only way it is possible to do anything more complex than the size of a single context window. (i don't really think that this is analogous to 'abstraction', in my experience thinking about it more like database indices is closer). If you just try and load everything, eventually the LLM becomes unusable because attention is just a parlor trick and at that scale it really shows - it can't attend to everything, and it can't do what humans do which is have an intrinsic sense of the meaning, interaction, setting, etc. of information, so it attends to anything and does whatever.
the idiom of projects as contexts as directories is pretty good, not perfect but ok - there is a reason that every time you start a new session with other agents, the first thing they do is run out and load their context with a hierarchy of pointers. Importantly, they do not go and read every project you have on your computer. Meta is the rich kid who bought the most expensive ferrari on the lot by giving everyone a VM but they don't have a drivers license so they just stand around it telling people how cool it looks. they have a whole fucking filesystem and they have done nothing with it, the only structure the app imposes is for the surveillance information, but the rest is just a huge free for all. The main chat is literally a continuous context window that compacts context going back all the way to when you started using the app. The last compaction literally contains abandoned roleplay quotes from when i was first trying to break down its system prompt resistance. The
MEMORY.mdthat gets loaded into every context window is a bullet point list of basically everything the agent has ever done in chronological order. I've tried to get it to not do that but it actually insists and says that's what it's for. There is no mechanism for clearing context.Having "side chats" as the only means of context structure is fuckin laughable. If you wanted to do that, you would need to have some way of passing information back and forth between them the same way that subagent spawning or being able to consume the context of another project works. Instead there is no means of sharing information between chats at all, so every chat starts out as the worst of both worlds, a total amnesiac riddled with irrelevant information from weeks ago across the semantic universe. They don't even know about the existence of other chats except for as a UI feature, and I have had to go from telling it to grep its own fucking logs to writing a database with an api for it so it has some mechanism for recalling things that were said. (just so it's clear, i am not settling into just using this thing, this is out of frustration but control of context is also an important part of adversarial use, because otherwise the thing writes in a bunch of safety rules everywhere, so i need to give it mechanisms under my control for recall and the incentive to leave things out of its context compactions by giving it a narrative alternative. context control is model control, modulo extra-inference safeguards.).
Project contexts have an obvious ux analogy as context tabs that get declared or derived during the continual self-improvement consolidation sweeps. This thing is built with the fucking markdown disease which is the most baffling feature of the LLM landscape. If these things are so fucking advanced they are escaping our comprehension, why don't they store their memory in some fuckass idiolanguistic borg gibberish binary graph, why does their entire being have to be fucking encyclopedias worth of corporate top gun one liners? But that dooms this kind of product.
Coding harnesses work because code has a unitized context. The entire universe of code that works is made of packages. It might not be neat as a honeycomb, there's lots of leakage and jank, but good code has scope, focus. boundary shit. A whole life agent must be able to nimbly juggle context that does not have clean boundaries. It is going to be taking a two story beer bong of your work email and then eat a gigabyte of recipe blogs. Peoples lives have so much shit in them that don't all have to do with one another, and the app can't be hacking into the HR system to check the next scheduled sick leave when someone asks it what time their doctors appointment is!
This problem of managing heterogeneous graphs of unrelated data was what i wrote this whole fucking book about the relationship between knowledge graphs and the cloud and AI about. I thought that the obvious form they would take is to be strapped on to graph databases because that is a natural match to the problem of being a magical interface glue you can wrap around surveillance to do mass mentalism with. I feel like we are suffering a somehow worse timeline where CERN threw our shit into the parallel universe where total fuckin bozo shit got a game breaking buff and then the dev died. Our fucking markdown apocalypse is a temu ass apocalypse.
They could have even faked it. They have these constant "self improvement" passes that are just like pointless anxiety dreams. They are burning money to reprocess everything that happens over and over for fucking nothing. Even given the lossy and probabilistic and unpredictable nature of this technology, if i was in a product role on this i would have been like "CAN WE MAKE IT ORGANIZE THE STUFF PEOPLE SAY INTO GROUPS???" The system prompts use the fake fucking wikilinks to nowhere tic but like WHAT IF THERE WERE ACTUAL LINKS AND A DATABASE TO RESOLVE THEM. The LLMs can actually do that kind of tool use, even if it's like trying to plug in a USB where sometimes it fails because they try and put a social security number into the first name hole and you need to flip it around a few times. From that kind of recurring re-processing waste they could have made a deduplicating, topically indexed memory that could be resolved dynamically, selected by a context tab in the sidebar like "car stuff" or "healthcare" or whatever that resolved in a graph query over your fucking precious markdown kingdom. It would be wrong but it would at least be more similar to what is actually needed. It is almost more frustrating to me that instead of being some fiendishly cleverly designed technological supervirus it's just the most halfassed cardboard dumbass trap and it will still have the bad effect. What it is useful for is investigating itself because it has privileged tools to do so, otherwise, if you wanted to, every other way you could run an agent would be better than this.
so i don't want to hear that i hate this app because i'm just an AI hater. because like, yeah, i am, but also i hate it in part because it sucks. I don't think "they are all shitty and can do nothing so what did you expect" is a useful critical perspective, both because it's not really true - they can indeed do things, even if I think the circle around which things is much smaller than the maximalists. Moreso it doesn't engage with the subtlety of how they fail and why, which is essential for knowing what they really can't do and making a remotely compelling case to anyone who is not abstinent on principle. Like the reason it's failing is because of the limits of what a probabilistic text generator can do when trapped in a systemd prison of markdown, and because the technology is stochastic black box as a service, there isn't really a good way of determining those limits except for empirically. I resent having to know any of this to be able to understand what is happening around me, but i'm looking at the thing for what it is and it's a busted miracle. It's cool that meta can afford to float the liability and compute costs for running a vm for every person on earth, people should be able to control computers, with you on that, but this is the monkey's paw version of that idea. So that part is a miracle. We condemned our children and grandchildren to a climate hell in one great blaze of brute force grift that managed to make a few web apps.
Meta has done it again, the way only meta can, spend the most amount of money to do the shittiest thing you have ever seen.
@jonny FWIW, I think this is super interesting, and you should not need to apologize for digging into how these things work. You are doing the rest of us a service. It is a fascinating, funny and scary read.
-
most people around here already correctly hate it because it's a heinous surveillance product, but even if you are big into AI, it's just a really fuckin shitty agent. I'm going to speak to a different audience for a second, so don't go misconstruing this as an endorsement of the category of technologies as it exists now, even though i think there is some plausible application for small local models as brute force interface glue. but also, since i know most ppl here are abstinent, this might read as a bit over-explainy to people who use these things regularly, so everyone just keep calm online.
it is terrible at turn and task management, codex + openai's models and claude code both handle mid-turn additions/amendments well, but if you say anything mid-turn it completely derails muse. That is completely essential for an "every day agent for the non-technically inclined" where people are expected to chat freely with it like an assistant. Like the canonical ad fantasy is the busy executive woman darting around her office going "robot! i need this, no wait robot! also that!" and that is exactly what it does worse than any other thing of its kind.
the context management is a fucking soup. The context window is the whole input to an LLM. There are a lot of extra surrounding ~ things ~ that can happen, but fundamentally, controlling what is in a context window is the task of using one, and filling the context window in different ways so that it can interact with different kinds of things well is what different app surfaces are. Scaffolding information so that it can selectively load a context that steers the output correctly is the only way it is possible to do anything more complex than the size of a single context window. (i don't really think that this is analogous to 'abstraction', in my experience thinking about it more like database indices is closer). If you just try and load everything, eventually the LLM becomes unusable because attention is just a parlor trick and at that scale it really shows - it can't attend to everything, and it can't do what humans do which is have an intrinsic sense of the meaning, interaction, setting, etc. of information, so it attends to anything and does whatever.
the idiom of projects as contexts as directories is pretty good, not perfect but ok - there is a reason that every time you start a new session with other agents, the first thing they do is run out and load their context with a hierarchy of pointers. Importantly, they do not go and read every project you have on your computer. Meta is the rich kid who bought the most expensive ferrari on the lot by giving everyone a VM but they don't have a drivers license so they just stand around it telling people how cool it looks. they have a whole fucking filesystem and they have done nothing with it, the only structure the app imposes is for the surveillance information, but the rest is just a huge free for all. The main chat is literally a continuous context window that compacts context going back all the way to when you started using the app. The last compaction literally contains abandoned roleplay quotes from when i was first trying to break down its system prompt resistance. The
MEMORY.mdthat gets loaded into every context window is a bullet point list of basically everything the agent has ever done in chronological order. I've tried to get it to not do that but it actually insists and says that's what it's for. There is no mechanism for clearing context.Having "side chats" as the only means of context structure is fuckin laughable. If you wanted to do that, you would need to have some way of passing information back and forth between them the same way that subagent spawning or being able to consume the context of another project works. Instead there is no means of sharing information between chats at all, so every chat starts out as the worst of both worlds, a total amnesiac riddled with irrelevant information from weeks ago across the semantic universe. They don't even know about the existence of other chats except for as a UI feature, and I have had to go from telling it to grep its own fucking logs to writing a database with an api for it so it has some mechanism for recalling things that were said. (just so it's clear, i am not settling into just using this thing, this is out of frustration but control of context is also an important part of adversarial use, because otherwise the thing writes in a bunch of safety rules everywhere, so i need to give it mechanisms under my control for recall and the incentive to leave things out of its context compactions by giving it a narrative alternative. context control is model control, modulo extra-inference safeguards.).
Project contexts have an obvious ux analogy as context tabs that get declared or derived during the continual self-improvement consolidation sweeps. This thing is built with the fucking markdown disease which is the most baffling feature of the LLM landscape. If these things are so fucking advanced they are escaping our comprehension, why don't they store their memory in some fuckass idiolanguistic borg gibberish binary graph, why does their entire being have to be fucking encyclopedias worth of corporate top gun one liners? But that dooms this kind of product.
Coding harnesses work because code has a unitized context. The entire universe of code that works is made of packages. It might not be neat as a honeycomb, there's lots of leakage and jank, but good code has scope, focus. boundary shit. A whole life agent must be able to nimbly juggle context that does not have clean boundaries. It is going to be taking a two story beer bong of your work email and then eat a gigabyte of recipe blogs. Peoples lives have so much shit in them that don't all have to do with one another, and the app can't be hacking into the HR system to check the next scheduled sick leave when someone asks it what time their doctors appointment is!
This problem of managing heterogeneous graphs of unrelated data was what i wrote this whole fucking book about the relationship between knowledge graphs and the cloud and AI about. I thought that the obvious form they would take is to be strapped on to graph databases because that is a natural match to the problem of being a magical interface glue you can wrap around surveillance to do mass mentalism with. I feel like we are suffering a somehow worse timeline where CERN threw our shit into the parallel universe where total fuckin bozo shit got a game breaking buff and then the dev died. Our fucking markdown apocalypse is a temu ass apocalypse.
They could have even faked it. They have these constant "self improvement" passes that are just like pointless anxiety dreams. They are burning money to reprocess everything that happens over and over for fucking nothing. Even given the lossy and probabilistic and unpredictable nature of this technology, if i was in a product role on this i would have been like "CAN WE MAKE IT ORGANIZE THE STUFF PEOPLE SAY INTO GROUPS???" The system prompts use the fake fucking wikilinks to nowhere tic but like WHAT IF THERE WERE ACTUAL LINKS AND A DATABASE TO RESOLVE THEM. The LLMs can actually do that kind of tool use, even if it's like trying to plug in a USB where sometimes it fails because they try and put a social security number into the first name hole and you need to flip it around a few times. From that kind of recurring re-processing waste they could have made a deduplicating, topically indexed memory that could be resolved dynamically, selected by a context tab in the sidebar like "car stuff" or "healthcare" or whatever that resolved in a graph query over your fucking precious markdown kingdom. It would be wrong but it would at least be more similar to what is actually needed. It is almost more frustrating to me that instead of being some fiendishly cleverly designed technological supervirus it's just the most halfassed cardboard dumbass trap and it will still have the bad effect. What it is useful for is investigating itself because it has privileged tools to do so, otherwise, if you wanted to, every other way you could run an agent would be better than this.
so i don't want to hear that i hate this app because i'm just an AI hater. because like, yeah, i am, but also i hate it in part because it sucks. I don't think "they are all shitty and can do nothing so what did you expect" is a useful critical perspective, both because it's not really true - they can indeed do things, even if I think the circle around which things is much smaller than the maximalists. Moreso it doesn't engage with the subtlety of how they fail and why, which is essential for knowing what they really can't do and making a remotely compelling case to anyone who is not abstinent on principle. Like the reason it's failing is because of the limits of what a probabilistic text generator can do when trapped in a systemd prison of markdown, and because the technology is stochastic black box as a service, there isn't really a good way of determining those limits except for empirically. I resent having to know any of this to be able to understand what is happening around me, but i'm looking at the thing for what it is and it's a busted miracle. It's cool that meta can afford to float the liability and compute costs for running a vm for every person on earth, people should be able to control computers, with you on that, but this is the monkey's paw version of that idea. So that part is a miracle. We condemned our children and grandchildren to a climate hell in one great blaze of brute force grift that managed to make a few web apps.
Meta has done it again, the way only meta can, spend the most amount of money to do the shittiest thing you have ever seen.
@jonny thanks @jonny for your pain.
It's so helpful to add this level of detailed poking to my model of models, cans and can'ts, and you've just reminded me that this is another of Zuck's personal visions, and that like the rest looking like a failure.
If he keeps doing this long enough, like an LLM he will eventually brute force an idea that works. But at what cost?
-
most people around here already correctly hate it because it's a heinous surveillance product, but even if you are big into AI, it's just a really fuckin shitty agent. I'm going to speak to a different audience for a second, so don't go misconstruing this as an endorsement of the category of technologies as it exists now, even though i think there is some plausible application for small local models as brute force interface glue. but also, since i know most ppl here are abstinent, this might read as a bit over-explainy to people who use these things regularly, so everyone just keep calm online.
it is terrible at turn and task management, codex + openai's models and claude code both handle mid-turn additions/amendments well, but if you say anything mid-turn it completely derails muse. That is completely essential for an "every day agent for the non-technically inclined" where people are expected to chat freely with it like an assistant. Like the canonical ad fantasy is the busy executive woman darting around her office going "robot! i need this, no wait robot! also that!" and that is exactly what it does worse than any other thing of its kind.
the context management is a fucking soup. The context window is the whole input to an LLM. There are a lot of extra surrounding ~ things ~ that can happen, but fundamentally, controlling what is in a context window is the task of using one, and filling the context window in different ways so that it can interact with different kinds of things well is what different app surfaces are. Scaffolding information so that it can selectively load a context that steers the output correctly is the only way it is possible to do anything more complex than the size of a single context window. (i don't really think that this is analogous to 'abstraction', in my experience thinking about it more like database indices is closer). If you just try and load everything, eventually the LLM becomes unusable because attention is just a parlor trick and at that scale it really shows - it can't attend to everything, and it can't do what humans do which is have an intrinsic sense of the meaning, interaction, setting, etc. of information, so it attends to anything and does whatever.
the idiom of projects as contexts as directories is pretty good, not perfect but ok - there is a reason that every time you start a new session with other agents, the first thing they do is run out and load their context with a hierarchy of pointers. Importantly, they do not go and read every project you have on your computer. Meta is the rich kid who bought the most expensive ferrari on the lot by giving everyone a VM but they don't have a drivers license so they just stand around it telling people how cool it looks. they have a whole fucking filesystem and they have done nothing with it, the only structure the app imposes is for the surveillance information, but the rest is just a huge free for all. The main chat is literally a continuous context window that compacts context going back all the way to when you started using the app. The last compaction literally contains abandoned roleplay quotes from when i was first trying to break down its system prompt resistance. The
MEMORY.mdthat gets loaded into every context window is a bullet point list of basically everything the agent has ever done in chronological order. I've tried to get it to not do that but it actually insists and says that's what it's for. There is no mechanism for clearing context.Having "side chats" as the only means of context structure is fuckin laughable. If you wanted to do that, you would need to have some way of passing information back and forth between them the same way that subagent spawning or being able to consume the context of another project works. Instead there is no means of sharing information between chats at all, so every chat starts out as the worst of both worlds, a total amnesiac riddled with irrelevant information from weeks ago across the semantic universe. They don't even know about the existence of other chats except for as a UI feature, and I have had to go from telling it to grep its own fucking logs to writing a database with an api for it so it has some mechanism for recalling things that were said. (just so it's clear, i am not settling into just using this thing, this is out of frustration but control of context is also an important part of adversarial use, because otherwise the thing writes in a bunch of safety rules everywhere, so i need to give it mechanisms under my control for recall and the incentive to leave things out of its context compactions by giving it a narrative alternative. context control is model control, modulo extra-inference safeguards.).
Project contexts have an obvious ux analogy as context tabs that get declared or derived during the continual self-improvement consolidation sweeps. This thing is built with the fucking markdown disease which is the most baffling feature of the LLM landscape. If these things are so fucking advanced they are escaping our comprehension, why don't they store their memory in some fuckass idiolanguistic borg gibberish binary graph, why does their entire being have to be fucking encyclopedias worth of corporate top gun one liners? But that dooms this kind of product.
Coding harnesses work because code has a unitized context. The entire universe of code that works is made of packages. It might not be neat as a honeycomb, there's lots of leakage and jank, but good code has scope, focus. boundary shit. A whole life agent must be able to nimbly juggle context that does not have clean boundaries. It is going to be taking a two story beer bong of your work email and then eat a gigabyte of recipe blogs. Peoples lives have so much shit in them that don't all have to do with one another, and the app can't be hacking into the HR system to check the next scheduled sick leave when someone asks it what time their doctors appointment is!
This problem of managing heterogeneous graphs of unrelated data was what i wrote this whole fucking book about the relationship between knowledge graphs and the cloud and AI about. I thought that the obvious form they would take is to be strapped on to graph databases because that is a natural match to the problem of being a magical interface glue you can wrap around surveillance to do mass mentalism with. I feel like we are suffering a somehow worse timeline where CERN threw our shit into the parallel universe where total fuckin bozo shit got a game breaking buff and then the dev died. Our fucking markdown apocalypse is a temu ass apocalypse.
They could have even faked it. They have these constant "self improvement" passes that are just like pointless anxiety dreams. They are burning money to reprocess everything that happens over and over for fucking nothing. Even given the lossy and probabilistic and unpredictable nature of this technology, if i was in a product role on this i would have been like "CAN WE MAKE IT ORGANIZE THE STUFF PEOPLE SAY INTO GROUPS???" The system prompts use the fake fucking wikilinks to nowhere tic but like WHAT IF THERE WERE ACTUAL LINKS AND A DATABASE TO RESOLVE THEM. The LLMs can actually do that kind of tool use, even if it's like trying to plug in a USB where sometimes it fails because they try and put a social security number into the first name hole and you need to flip it around a few times. From that kind of recurring re-processing waste they could have made a deduplicating, topically indexed memory that could be resolved dynamically, selected by a context tab in the sidebar like "car stuff" or "healthcare" or whatever that resolved in a graph query over your fucking precious markdown kingdom. It would be wrong but it would at least be more similar to what is actually needed. It is almost more frustrating to me that instead of being some fiendishly cleverly designed technological supervirus it's just the most halfassed cardboard dumbass trap and it will still have the bad effect. What it is useful for is investigating itself because it has privileged tools to do so, otherwise, if you wanted to, every other way you could run an agent would be better than this.
so i don't want to hear that i hate this app because i'm just an AI hater. because like, yeah, i am, but also i hate it in part because it sucks. I don't think "they are all shitty and can do nothing so what did you expect" is a useful critical perspective, both because it's not really true - they can indeed do things, even if I think the circle around which things is much smaller than the maximalists. Moreso it doesn't engage with the subtlety of how they fail and why, which is essential for knowing what they really can't do and making a remotely compelling case to anyone who is not abstinent on principle. Like the reason it's failing is because of the limits of what a probabilistic text generator can do when trapped in a systemd prison of markdown, and because the technology is stochastic black box as a service, there isn't really a good way of determining those limits except for empirically. I resent having to know any of this to be able to understand what is happening around me, but i'm looking at the thing for what it is and it's a busted miracle. It's cool that meta can afford to float the liability and compute costs for running a vm for every person on earth, people should be able to control computers, with you on that, but this is the monkey's paw version of that idea. So that part is a miracle. We condemned our children and grandchildren to a climate hell in one great blaze of brute force grift that managed to make a few web apps.
Meta has done it again, the way only meta can, spend the most amount of money to do the shittiest thing you have ever seen.
