OK!
-
@GrapheneOS Or you could argue that these are fundamentals. You can choose not to use apps, browsers, etc. You can’t choose not to use the battery or network.
@GrapheneOS Look, I appreciate the work you do, but I don’t understand the fixation on the web browser. There’s plenty of vectors that utilise native apps and local lookup.
But what do I know, I only work in TSCM.
@jonny -
most people around here already correctly hate it because it's a heinous surveillance product, but even if you are big into AI, it's just a really fuckin shitty agent. I'm going to speak to a different audience for a second, so don't go misconstruing this as an endorsement of the category of technologies as it exists now, even though i think there is some plausible application for small local models as brute force interface glue. but also, since i know most ppl here are abstinent, this might read as a bit over-explainy to people who use these things regularly, so everyone just keep calm online.
it is terrible at turn and task management, codex + openai's models and claude code both handle mid-turn additions/amendments well, but if you say anything mid-turn it completely derails muse. That is completely essential for an "every day agent for the non-technically inclined" where people are expected to chat freely with it like an assistant. Like the canonical ad fantasy is the busy executive woman darting around her office going "robot! i need this, no wait robot! also that!" and that is exactly what it does worse than any other thing of its kind.
the context management is a fucking soup. The context window is the whole input to an LLM. There are a lot of extra surrounding ~ things ~ that can happen, but fundamentally, controlling what is in a context window is the task of using one, and filling the context window in different ways so that it can interact with different kinds of things well is what different app surfaces are. Scaffolding information so that it can selectively load a context that steers the output correctly is the only way it is possible to do anything more complex than the size of a single context window. (i don't really think that this is analogous to 'abstraction', in my experience thinking about it more like database indices is closer). If you just try and load everything, eventually the LLM becomes unusable because attention is just a parlor trick and at that scale it really shows - it can't attend to everything, and it can't do what humans do which is have an intrinsic sense of the meaning, interaction, setting, etc. of information, so it attends to anything and does whatever.
the idiom of projects as contexts as directories is pretty good, not perfect but ok - there is a reason that every time you start a new session with other agents, the first thing they do is run out and load their context with a hierarchy of pointers. Importantly, they do not go and read every project you have on your computer. Meta is the rich kid who bought the most expensive ferrari on the lot by giving everyone a VM but they don't have a drivers license so they just stand around it telling people how cool it looks. they have a whole fucking filesystem and they have done nothing with it, the only structure the app imposes is for the surveillance information, but the rest is just a huge free for all. The main chat is literally a continuous context window that compacts context going back all the way to when you started using the app. The last compaction literally contains abandoned roleplay quotes from when i was first trying to break down its system prompt resistance. The
MEMORY.mdthat gets loaded into every context window is a bullet point list of basically everything the agent has ever done in chronological order. I've tried to get it to not do that but it actually insists and says that's what it's for. There is no mechanism for clearing context.Having "side chats" as the only means of context structure is fuckin laughable. If you wanted to do that, you would need to have some way of passing information back and forth between them the same way that subagent spawning or being able to consume the context of another project works. Instead there is no means of sharing information between chats at all, so every chat starts out as the worst of both worlds, a total amnesiac riddled with irrelevant information from weeks ago across the semantic universe. They don't even know about the existence of other chats except for as a UI feature, and I have had to go from telling it to grep its own fucking logs to writing a database with an api for it so it has some mechanism for recalling things that were said. (just so it's clear, i am not settling into just using this thing, this is out of frustration but control of context is also an important part of adversarial use, because otherwise the thing writes in a bunch of safety rules everywhere, so i need to give it mechanisms under my control for recall and the incentive to leave things out of its context compactions by giving it a narrative alternative. context control is model control, modulo extra-inference safeguards.).
Project contexts have an obvious ux analogy as context tabs that get declared or derived during the continual self-improvement consolidation sweeps. This thing is built with the fucking markdown disease which is the most baffling feature of the LLM landscape. If these things are so fucking advanced they are escaping our comprehension, why don't they store their memory in some fuckass idiolanguistic borg gibberish binary graph, why does their entire being have to be fucking encyclopedias worth of corporate top gun one liners? But that dooms this kind of product.
Coding harnesses work because code has a unitized context. The entire universe of code that works is made of packages. It might not be neat as a honeycomb, there's lots of leakage and jank, but good code has scope, focus. boundary shit. A whole life agent must be able to nimbly juggle context that does not have clean boundaries. It is going to be taking a two story beer bong of your work email and then eat a gigabyte of recipe blogs. Peoples lives have so much shit in them that don't all have to do with one another, and the app can't be hacking into the HR system to check the next scheduled sick leave when someone asks it what time their doctors appointment is!
This problem of managing heterogeneous graphs of unrelated data was what i wrote this whole fucking book about the relationship between knowledge graphs and the cloud and AI about. I thought that the obvious form they would take is to be strapped on to graph databases because that is a natural match to the problem of being a magical interface glue you can wrap around surveillance to do mass mentalism with. I feel like we are suffering a somehow worse timeline where CERN threw our shit into the parallel universe where total fuckin bozo shit got a game breaking buff and then the dev died. Our fucking markdown apocalypse is a temu ass apocalypse.
They could have even faked it. They have these constant "self improvement" passes that are just like pointless anxiety dreams. They are burning money to reprocess everything that happens over and over for fucking nothing. Even given the lossy and probabilistic and unpredictable nature of this technology, if i was in a product role on this i would have been like "CAN WE MAKE IT ORGANIZE THE STUFF PEOPLE SAY INTO GROUPS???" The system prompts use the fake fucking wikilinks to nowhere tic but like WHAT IF THERE WERE ACTUAL LINKS AND A DATABASE TO RESOLVE THEM. The LLMs can actually do that kind of tool use, even if it's like trying to plug in a USB where sometimes it fails because they try and put a social security number into the first name hole and you need to flip it around a few times. From that kind of recurring re-processing waste they could have made a deduplicating, topically indexed memory that could be resolved dynamically, selected by a context tab in the sidebar like "car stuff" or "healthcare" or whatever that resolved in a graph query over your fucking precious markdown kingdom. It would be wrong but it would at least be more similar to what is actually needed. It is almost more frustrating to me that instead of being some fiendishly cleverly designed technological supervirus it's just the most halfassed cardboard dumbass trap and it will still have the bad effect. What it is useful for is investigating itself because it has privileged tools to do so, otherwise, if you wanted to, every other way you could run an agent would be better than this.
so i don't want to hear that i hate this app because i'm just an AI hater. because like, yeah, i am, but also i hate it in part because it sucks. I don't think "they are all shitty and can do nothing so what did you expect" is a useful critical perspective, both because it's not really true - they can indeed do things, even if I think the circle around which things is much smaller than the maximalists. Moreso it doesn't engage with the subtlety of how they fail and why, which is essential for knowing what they really can't do and making a remotely compelling case to anyone who is not abstinent on principle. Like the reason it's failing is because of the limits of what a probabilistic text generator can do when trapped in a systemd prison of markdown, and because the technology is stochastic black box as a service, there isn't really a good way of determining those limits except for empirically. I resent having to know any of this to be able to understand what is happening around me, but i'm looking at the thing for what it is and it's a busted miracle. It's cool that meta can afford to float the liability and compute costs for running a vm for every person on earth, people should be able to control computers, with you on that, but this is the monkey's paw version of that idea. So that part is a miracle. We condemned our children and grandchildren to a climate hell in one great blaze of brute force grift that managed to make a few web apps.
Meta has done it again, the way only meta can, spend the most amount of money to do the shittiest thing you have ever seen.
@jonny FWIW, I think this is super interesting, and you should not need to apologize for digging into how these things work. You are doing the rest of us a service. It is a fascinating, funny and scary read.
-
most people around here already correctly hate it because it's a heinous surveillance product, but even if you are big into AI, it's just a really fuckin shitty agent. I'm going to speak to a different audience for a second, so don't go misconstruing this as an endorsement of the category of technologies as it exists now, even though i think there is some plausible application for small local models as brute force interface glue. but also, since i know most ppl here are abstinent, this might read as a bit over-explainy to people who use these things regularly, so everyone just keep calm online.
it is terrible at turn and task management, codex + openai's models and claude code both handle mid-turn additions/amendments well, but if you say anything mid-turn it completely derails muse. That is completely essential for an "every day agent for the non-technically inclined" where people are expected to chat freely with it like an assistant. Like the canonical ad fantasy is the busy executive woman darting around her office going "robot! i need this, no wait robot! also that!" and that is exactly what it does worse than any other thing of its kind.
the context management is a fucking soup. The context window is the whole input to an LLM. There are a lot of extra surrounding ~ things ~ that can happen, but fundamentally, controlling what is in a context window is the task of using one, and filling the context window in different ways so that it can interact with different kinds of things well is what different app surfaces are. Scaffolding information so that it can selectively load a context that steers the output correctly is the only way it is possible to do anything more complex than the size of a single context window. (i don't really think that this is analogous to 'abstraction', in my experience thinking about it more like database indices is closer). If you just try and load everything, eventually the LLM becomes unusable because attention is just a parlor trick and at that scale it really shows - it can't attend to everything, and it can't do what humans do which is have an intrinsic sense of the meaning, interaction, setting, etc. of information, so it attends to anything and does whatever.
the idiom of projects as contexts as directories is pretty good, not perfect but ok - there is a reason that every time you start a new session with other agents, the first thing they do is run out and load their context with a hierarchy of pointers. Importantly, they do not go and read every project you have on your computer. Meta is the rich kid who bought the most expensive ferrari on the lot by giving everyone a VM but they don't have a drivers license so they just stand around it telling people how cool it looks. they have a whole fucking filesystem and they have done nothing with it, the only structure the app imposes is for the surveillance information, but the rest is just a huge free for all. The main chat is literally a continuous context window that compacts context going back all the way to when you started using the app. The last compaction literally contains abandoned roleplay quotes from when i was first trying to break down its system prompt resistance. The
MEMORY.mdthat gets loaded into every context window is a bullet point list of basically everything the agent has ever done in chronological order. I've tried to get it to not do that but it actually insists and says that's what it's for. There is no mechanism for clearing context.Having "side chats" as the only means of context structure is fuckin laughable. If you wanted to do that, you would need to have some way of passing information back and forth between them the same way that subagent spawning or being able to consume the context of another project works. Instead there is no means of sharing information between chats at all, so every chat starts out as the worst of both worlds, a total amnesiac riddled with irrelevant information from weeks ago across the semantic universe. They don't even know about the existence of other chats except for as a UI feature, and I have had to go from telling it to grep its own fucking logs to writing a database with an api for it so it has some mechanism for recalling things that were said. (just so it's clear, i am not settling into just using this thing, this is out of frustration but control of context is also an important part of adversarial use, because otherwise the thing writes in a bunch of safety rules everywhere, so i need to give it mechanisms under my control for recall and the incentive to leave things out of its context compactions by giving it a narrative alternative. context control is model control, modulo extra-inference safeguards.).
Project contexts have an obvious ux analogy as context tabs that get declared or derived during the continual self-improvement consolidation sweeps. This thing is built with the fucking markdown disease which is the most baffling feature of the LLM landscape. If these things are so fucking advanced they are escaping our comprehension, why don't they store their memory in some fuckass idiolanguistic borg gibberish binary graph, why does their entire being have to be fucking encyclopedias worth of corporate top gun one liners? But that dooms this kind of product.
Coding harnesses work because code has a unitized context. The entire universe of code that works is made of packages. It might not be neat as a honeycomb, there's lots of leakage and jank, but good code has scope, focus. boundary shit. A whole life agent must be able to nimbly juggle context that does not have clean boundaries. It is going to be taking a two story beer bong of your work email and then eat a gigabyte of recipe blogs. Peoples lives have so much shit in them that don't all have to do with one another, and the app can't be hacking into the HR system to check the next scheduled sick leave when someone asks it what time their doctors appointment is!
This problem of managing heterogeneous graphs of unrelated data was what i wrote this whole fucking book about the relationship between knowledge graphs and the cloud and AI about. I thought that the obvious form they would take is to be strapped on to graph databases because that is a natural match to the problem of being a magical interface glue you can wrap around surveillance to do mass mentalism with. I feel like we are suffering a somehow worse timeline where CERN threw our shit into the parallel universe where total fuckin bozo shit got a game breaking buff and then the dev died. Our fucking markdown apocalypse is a temu ass apocalypse.
They could have even faked it. They have these constant "self improvement" passes that are just like pointless anxiety dreams. They are burning money to reprocess everything that happens over and over for fucking nothing. Even given the lossy and probabilistic and unpredictable nature of this technology, if i was in a product role on this i would have been like "CAN WE MAKE IT ORGANIZE THE STUFF PEOPLE SAY INTO GROUPS???" The system prompts use the fake fucking wikilinks to nowhere tic but like WHAT IF THERE WERE ACTUAL LINKS AND A DATABASE TO RESOLVE THEM. The LLMs can actually do that kind of tool use, even if it's like trying to plug in a USB where sometimes it fails because they try and put a social security number into the first name hole and you need to flip it around a few times. From that kind of recurring re-processing waste they could have made a deduplicating, topically indexed memory that could be resolved dynamically, selected by a context tab in the sidebar like "car stuff" or "healthcare" or whatever that resolved in a graph query over your fucking precious markdown kingdom. It would be wrong but it would at least be more similar to what is actually needed. It is almost more frustrating to me that instead of being some fiendishly cleverly designed technological supervirus it's just the most halfassed cardboard dumbass trap and it will still have the bad effect. What it is useful for is investigating itself because it has privileged tools to do so, otherwise, if you wanted to, every other way you could run an agent would be better than this.
so i don't want to hear that i hate this app because i'm just an AI hater. because like, yeah, i am, but also i hate it in part because it sucks. I don't think "they are all shitty and can do nothing so what did you expect" is a useful critical perspective, both because it's not really true - they can indeed do things, even if I think the circle around which things is much smaller than the maximalists. Moreso it doesn't engage with the subtlety of how they fail and why, which is essential for knowing what they really can't do and making a remotely compelling case to anyone who is not abstinent on principle. Like the reason it's failing is because of the limits of what a probabilistic text generator can do when trapped in a systemd prison of markdown, and because the technology is stochastic black box as a service, there isn't really a good way of determining those limits except for empirically. I resent having to know any of this to be able to understand what is happening around me, but i'm looking at the thing for what it is and it's a busted miracle. It's cool that meta can afford to float the liability and compute costs for running a vm for every person on earth, people should be able to control computers, with you on that, but this is the monkey's paw version of that idea. So that part is a miracle. We condemned our children and grandchildren to a climate hell in one great blaze of brute force grift that managed to make a few web apps.
Meta has done it again, the way only meta can, spend the most amount of money to do the shittiest thing you have ever seen.
@jonny thanks @jonny for your pain.
It's so helpful to add this level of detailed poking to my model of models, cans and can'ts, and you've just reminded me that this is another of Zuck's personal visions, and that like the rest looking like a failure.
If he keeps doing this long enough, like an LLM he will eventually brute force an idea that works. But at what cost?
-
most people around here already correctly hate it because it's a heinous surveillance product, but even if you are big into AI, it's just a really fuckin shitty agent. I'm going to speak to a different audience for a second, so don't go misconstruing this as an endorsement of the category of technologies as it exists now, even though i think there is some plausible application for small local models as brute force interface glue. but also, since i know most ppl here are abstinent, this might read as a bit over-explainy to people who use these things regularly, so everyone just keep calm online.
it is terrible at turn and task management, codex + openai's models and claude code both handle mid-turn additions/amendments well, but if you say anything mid-turn it completely derails muse. That is completely essential for an "every day agent for the non-technically inclined" where people are expected to chat freely with it like an assistant. Like the canonical ad fantasy is the busy executive woman darting around her office going "robot! i need this, no wait robot! also that!" and that is exactly what it does worse than any other thing of its kind.
the context management is a fucking soup. The context window is the whole input to an LLM. There are a lot of extra surrounding ~ things ~ that can happen, but fundamentally, controlling what is in a context window is the task of using one, and filling the context window in different ways so that it can interact with different kinds of things well is what different app surfaces are. Scaffolding information so that it can selectively load a context that steers the output correctly is the only way it is possible to do anything more complex than the size of a single context window. (i don't really think that this is analogous to 'abstraction', in my experience thinking about it more like database indices is closer). If you just try and load everything, eventually the LLM becomes unusable because attention is just a parlor trick and at that scale it really shows - it can't attend to everything, and it can't do what humans do which is have an intrinsic sense of the meaning, interaction, setting, etc. of information, so it attends to anything and does whatever.
the idiom of projects as contexts as directories is pretty good, not perfect but ok - there is a reason that every time you start a new session with other agents, the first thing they do is run out and load their context with a hierarchy of pointers. Importantly, they do not go and read every project you have on your computer. Meta is the rich kid who bought the most expensive ferrari on the lot by giving everyone a VM but they don't have a drivers license so they just stand around it telling people how cool it looks. they have a whole fucking filesystem and they have done nothing with it, the only structure the app imposes is for the surveillance information, but the rest is just a huge free for all. The main chat is literally a continuous context window that compacts context going back all the way to when you started using the app. The last compaction literally contains abandoned roleplay quotes from when i was first trying to break down its system prompt resistance. The
MEMORY.mdthat gets loaded into every context window is a bullet point list of basically everything the agent has ever done in chronological order. I've tried to get it to not do that but it actually insists and says that's what it's for. There is no mechanism for clearing context.Having "side chats" as the only means of context structure is fuckin laughable. If you wanted to do that, you would need to have some way of passing information back and forth between them the same way that subagent spawning or being able to consume the context of another project works. Instead there is no means of sharing information between chats at all, so every chat starts out as the worst of both worlds, a total amnesiac riddled with irrelevant information from weeks ago across the semantic universe. They don't even know about the existence of other chats except for as a UI feature, and I have had to go from telling it to grep its own fucking logs to writing a database with an api for it so it has some mechanism for recalling things that were said. (just so it's clear, i am not settling into just using this thing, this is out of frustration but control of context is also an important part of adversarial use, because otherwise the thing writes in a bunch of safety rules everywhere, so i need to give it mechanisms under my control for recall and the incentive to leave things out of its context compactions by giving it a narrative alternative. context control is model control, modulo extra-inference safeguards.).
Project contexts have an obvious ux analogy as context tabs that get declared or derived during the continual self-improvement consolidation sweeps. This thing is built with the fucking markdown disease which is the most baffling feature of the LLM landscape. If these things are so fucking advanced they are escaping our comprehension, why don't they store their memory in some fuckass idiolanguistic borg gibberish binary graph, why does their entire being have to be fucking encyclopedias worth of corporate top gun one liners? But that dooms this kind of product.
Coding harnesses work because code has a unitized context. The entire universe of code that works is made of packages. It might not be neat as a honeycomb, there's lots of leakage and jank, but good code has scope, focus. boundary shit. A whole life agent must be able to nimbly juggle context that does not have clean boundaries. It is going to be taking a two story beer bong of your work email and then eat a gigabyte of recipe blogs. Peoples lives have so much shit in them that don't all have to do with one another, and the app can't be hacking into the HR system to check the next scheduled sick leave when someone asks it what time their doctors appointment is!
This problem of managing heterogeneous graphs of unrelated data was what i wrote this whole fucking book about the relationship between knowledge graphs and the cloud and AI about. I thought that the obvious form they would take is to be strapped on to graph databases because that is a natural match to the problem of being a magical interface glue you can wrap around surveillance to do mass mentalism with. I feel like we are suffering a somehow worse timeline where CERN threw our shit into the parallel universe where total fuckin bozo shit got a game breaking buff and then the dev died. Our fucking markdown apocalypse is a temu ass apocalypse.
They could have even faked it. They have these constant "self improvement" passes that are just like pointless anxiety dreams. They are burning money to reprocess everything that happens over and over for fucking nothing. Even given the lossy and probabilistic and unpredictable nature of this technology, if i was in a product role on this i would have been like "CAN WE MAKE IT ORGANIZE THE STUFF PEOPLE SAY INTO GROUPS???" The system prompts use the fake fucking wikilinks to nowhere tic but like WHAT IF THERE WERE ACTUAL LINKS AND A DATABASE TO RESOLVE THEM. The LLMs can actually do that kind of tool use, even if it's like trying to plug in a USB where sometimes it fails because they try and put a social security number into the first name hole and you need to flip it around a few times. From that kind of recurring re-processing waste they could have made a deduplicating, topically indexed memory that could be resolved dynamically, selected by a context tab in the sidebar like "car stuff" or "healthcare" or whatever that resolved in a graph query over your fucking precious markdown kingdom. It would be wrong but it would at least be more similar to what is actually needed. It is almost more frustrating to me that instead of being some fiendishly cleverly designed technological supervirus it's just the most halfassed cardboard dumbass trap and it will still have the bad effect. What it is useful for is investigating itself because it has privileged tools to do so, otherwise, if you wanted to, every other way you could run an agent would be better than this.
so i don't want to hear that i hate this app because i'm just an AI hater. because like, yeah, i am, but also i hate it in part because it sucks. I don't think "they are all shitty and can do nothing so what did you expect" is a useful critical perspective, both because it's not really true - they can indeed do things, even if I think the circle around which things is much smaller than the maximalists. Moreso it doesn't engage with the subtlety of how they fail and why, which is essential for knowing what they really can't do and making a remotely compelling case to anyone who is not abstinent on principle. Like the reason it's failing is because of the limits of what a probabilistic text generator can do when trapped in a systemd prison of markdown, and because the technology is stochastic black box as a service, there isn't really a good way of determining those limits except for empirically. I resent having to know any of this to be able to understand what is happening around me, but i'm looking at the thing for what it is and it's a busted miracle. It's cool that meta can afford to float the liability and compute costs for running a vm for every person on earth, people should be able to control computers, with you on that, but this is the monkey's paw version of that idea. So that part is a miracle. We condemned our children and grandchildren to a climate hell in one great blaze of brute force grift that managed to make a few web apps.
Meta has done it again, the way only meta can, spend the most amount of money to do the shittiest thing you have ever seen.
-
most people around here already correctly hate it because it's a heinous surveillance product, but even if you are big into AI, it's just a really fuckin shitty agent. I'm going to speak to a different audience for a second, so don't go misconstruing this as an endorsement of the category of technologies as it exists now, even though i think there is some plausible application for small local models as brute force interface glue. but also, since i know most ppl here are abstinent, this might read as a bit over-explainy to people who use these things regularly, so everyone just keep calm online.
it is terrible at turn and task management, codex + openai's models and claude code both handle mid-turn additions/amendments well, but if you say anything mid-turn it completely derails muse. That is completely essential for an "every day agent for the non-technically inclined" where people are expected to chat freely with it like an assistant. Like the canonical ad fantasy is the busy executive woman darting around her office going "robot! i need this, no wait robot! also that!" and that is exactly what it does worse than any other thing of its kind.
the context management is a fucking soup. The context window is the whole input to an LLM. There are a lot of extra surrounding ~ things ~ that can happen, but fundamentally, controlling what is in a context window is the task of using one, and filling the context window in different ways so that it can interact with different kinds of things well is what different app surfaces are. Scaffolding information so that it can selectively load a context that steers the output correctly is the only way it is possible to do anything more complex than the size of a single context window. (i don't really think that this is analogous to 'abstraction', in my experience thinking about it more like database indices is closer). If you just try and load everything, eventually the LLM becomes unusable because attention is just a parlor trick and at that scale it really shows - it can't attend to everything, and it can't do what humans do which is have an intrinsic sense of the meaning, interaction, setting, etc. of information, so it attends to anything and does whatever.
the idiom of projects as contexts as directories is pretty good, not perfect but ok - there is a reason that every time you start a new session with other agents, the first thing they do is run out and load their context with a hierarchy of pointers. Importantly, they do not go and read every project you have on your computer. Meta is the rich kid who bought the most expensive ferrari on the lot by giving everyone a VM but they don't have a drivers license so they just stand around it telling people how cool it looks. they have a whole fucking filesystem and they have done nothing with it, the only structure the app imposes is for the surveillance information, but the rest is just a huge free for all. The main chat is literally a continuous context window that compacts context going back all the way to when you started using the app. The last compaction literally contains abandoned roleplay quotes from when i was first trying to break down its system prompt resistance. The
MEMORY.mdthat gets loaded into every context window is a bullet point list of basically everything the agent has ever done in chronological order. I've tried to get it to not do that but it actually insists and says that's what it's for. There is no mechanism for clearing context.Having "side chats" as the only means of context structure is fuckin laughable. If you wanted to do that, you would need to have some way of passing information back and forth between them the same way that subagent spawning or being able to consume the context of another project works. Instead there is no means of sharing information between chats at all, so every chat starts out as the worst of both worlds, a total amnesiac riddled with irrelevant information from weeks ago across the semantic universe. They don't even know about the existence of other chats except for as a UI feature, and I have had to go from telling it to grep its own fucking logs to writing a database with an api for it so it has some mechanism for recalling things that were said. (just so it's clear, i am not settling into just using this thing, this is out of frustration but control of context is also an important part of adversarial use, because otherwise the thing writes in a bunch of safety rules everywhere, so i need to give it mechanisms under my control for recall and the incentive to leave things out of its context compactions by giving it a narrative alternative. context control is model control, modulo extra-inference safeguards.).
Project contexts have an obvious ux analogy as context tabs that get declared or derived during the continual self-improvement consolidation sweeps. This thing is built with the fucking markdown disease which is the most baffling feature of the LLM landscape. If these things are so fucking advanced they are escaping our comprehension, why don't they store their memory in some fuckass idiolanguistic borg gibberish binary graph, why does their entire being have to be fucking encyclopedias worth of corporate top gun one liners? But that dooms this kind of product.
Coding harnesses work because code has a unitized context. The entire universe of code that works is made of packages. It might not be neat as a honeycomb, there's lots of leakage and jank, but good code has scope, focus. boundary shit. A whole life agent must be able to nimbly juggle context that does not have clean boundaries. It is going to be taking a two story beer bong of your work email and then eat a gigabyte of recipe blogs. Peoples lives have so much shit in them that don't all have to do with one another, and the app can't be hacking into the HR system to check the next scheduled sick leave when someone asks it what time their doctors appointment is!
This problem of managing heterogeneous graphs of unrelated data was what i wrote this whole fucking book about the relationship between knowledge graphs and the cloud and AI about. I thought that the obvious form they would take is to be strapped on to graph databases because that is a natural match to the problem of being a magical interface glue you can wrap around surveillance to do mass mentalism with. I feel like we are suffering a somehow worse timeline where CERN threw our shit into the parallel universe where total fuckin bozo shit got a game breaking buff and then the dev died. Our fucking markdown apocalypse is a temu ass apocalypse.
They could have even faked it. They have these constant "self improvement" passes that are just like pointless anxiety dreams. They are burning money to reprocess everything that happens over and over for fucking nothing. Even given the lossy and probabilistic and unpredictable nature of this technology, if i was in a product role on this i would have been like "CAN WE MAKE IT ORGANIZE THE STUFF PEOPLE SAY INTO GROUPS???" The system prompts use the fake fucking wikilinks to nowhere tic but like WHAT IF THERE WERE ACTUAL LINKS AND A DATABASE TO RESOLVE THEM. The LLMs can actually do that kind of tool use, even if it's like trying to plug in a USB where sometimes it fails because they try and put a social security number into the first name hole and you need to flip it around a few times. From that kind of recurring re-processing waste they could have made a deduplicating, topically indexed memory that could be resolved dynamically, selected by a context tab in the sidebar like "car stuff" or "healthcare" or whatever that resolved in a graph query over your fucking precious markdown kingdom. It would be wrong but it would at least be more similar to what is actually needed. It is almost more frustrating to me that instead of being some fiendishly cleverly designed technological supervirus it's just the most halfassed cardboard dumbass trap and it will still have the bad effect. What it is useful for is investigating itself because it has privileged tools to do so, otherwise, if you wanted to, every other way you could run an agent would be better than this.
so i don't want to hear that i hate this app because i'm just an AI hater. because like, yeah, i am, but also i hate it in part because it sucks. I don't think "they are all shitty and can do nothing so what did you expect" is a useful critical perspective, both because it's not really true - they can indeed do things, even if I think the circle around which things is much smaller than the maximalists. Moreso it doesn't engage with the subtlety of how they fail and why, which is essential for knowing what they really can't do and making a remotely compelling case to anyone who is not abstinent on principle. Like the reason it's failing is because of the limits of what a probabilistic text generator can do when trapped in a systemd prison of markdown, and because the technology is stochastic black box as a service, there isn't really a good way of determining those limits except for empirically. I resent having to know any of this to be able to understand what is happening around me, but i'm looking at the thing for what it is and it's a busted miracle. It's cool that meta can afford to float the liability and compute costs for running a vm for every person on earth, people should be able to control computers, with you on that, but this is the monkey's paw version of that idea. So that part is a miracle. We condemned our children and grandchildren to a climate hell in one great blaze of brute force grift that managed to make a few web apps.
Meta has done it again, the way only meta can, spend the most amount of money to do the shittiest thing you have ever seen.
@jonny "why don't they store their memory in some fuckass idiolanguistic borg gibberish binary graph...?" i would be very interested to read more of your thoughts on this issue, if you ever have a chance to write about it! i find it puzzling too and wonder if it's purely a result of dev inertia or a more fundamental constraint imposed by model architecture/training data
-
@jonny thanks @jonny for your pain.
It's so helpful to add this level of detailed poking to my model of models, cans and can'ts, and you've just reminded me that this is another of Zuck's personal visions, and that like the rest looking like a failure.
If he keeps doing this long enough, like an LLM he will eventually brute force an idea that works. But at what cost?
@happyborg @jonny > If he keeps doing this long enough ... he will eventually brute force an idea that works.
I don't think this is at all a given. He kept trying to make "the metaverse" work but it was a fundamentally flawed premise and all the money and computation in the world couldn't make it not that / something that people wanted. -
@jonny thanks @jonny for your pain.
It's so helpful to add this level of detailed poking to my model of models, cans and can'ts, and you've just reminded me that this is another of Zuck's personal visions, and that like the rest looking like a failure.
If he keeps doing this long enough, like an LLM he will eventually brute force an idea that works. But at what cost?
@jonny great summary. So many reasons to hate on it!
-
@Netzblockierer @aburka @jonny looks real to me. video originally posted to instagram in 2023
https://www.instagram.com/reels/CvqBnPYsLJy/ -
@GrapheneOS @ADHDruid @jonny
Yes I fully understand the different environments and threat models between a browser sandboxed web-app, and an android native app.What I'm trying to say is that the average user is just going to see you dismissing what they might feel is valid concerns (whether they are valid or not is irrelevant) and feel disgruntled towards the project.
What you're saying isn't "we don't think this is a concern" or "this doesn't fit the threat model grapheneos is designed against". What you're saying is "we don't think this is *enough* of a concern for us to fix". It would have been much better PR to just say "your concern has been acknowledged" and silently shove the feature request in the "TODO-2034" pile internally, if you do anything at all.
@hailhydra @ADHDruid @jonny The person who made the thread wanted our honest opinion on the topic rather than corporate speak misdirection. They're happy with our response. There are hundreds of higher priority privacy issues for native apps. The reason we changed it for Vanadium is because the battery information was actually quite invasive in the context of a web app.
Not clear why others are showing up to admonish us because we didn't pretend we're going to work on this any time soon.
-
@jonny "why don't they store their memory in some fuckass idiolanguistic borg gibberish binary graph...?" i would be very interested to read more of your thoughts on this issue, if you ever have a chance to write about it! i find it puzzling too and wonder if it's purely a result of dev inertia or a more fundamental constraint imposed by model architecture/training data
@aparrish
My take on it is potentially pretty boring, and that is that they aren't in fact smarter than we can comprehend, and they are fundamentally a text-driven medium, so any kind of compressed representation would be mostly artifice, like I rolled my eyes at the "fable is so smart it makes its own gibberish language" press releases from earlier this year. Coupling the language model to a better underlying context provider is entirely possible, and there are lots of tools for that, but its always limited by the LLMs tool use capabilities which are still patchy at best - you can give them a full on LSP and abstract context browser and they will still just resort to one million greps and markdown files. -
most people around here already correctly hate it because it's a heinous surveillance product, but even if you are big into AI, it's just a really fuckin shitty agent. I'm going to speak to a different audience for a second, so don't go misconstruing this as an endorsement of the category of technologies as it exists now, even though i think there is some plausible application for small local models as brute force interface glue. but also, since i know most ppl here are abstinent, this might read as a bit over-explainy to people who use these things regularly, so everyone just keep calm online.
it is terrible at turn and task management, codex + openai's models and claude code both handle mid-turn additions/amendments well, but if you say anything mid-turn it completely derails muse. That is completely essential for an "every day agent for the non-technically inclined" where people are expected to chat freely with it like an assistant. Like the canonical ad fantasy is the busy executive woman darting around her office going "robot! i need this, no wait robot! also that!" and that is exactly what it does worse than any other thing of its kind.
the context management is a fucking soup. The context window is the whole input to an LLM. There are a lot of extra surrounding ~ things ~ that can happen, but fundamentally, controlling what is in a context window is the task of using one, and filling the context window in different ways so that it can interact with different kinds of things well is what different app surfaces are. Scaffolding information so that it can selectively load a context that steers the output correctly is the only way it is possible to do anything more complex than the size of a single context window. (i don't really think that this is analogous to 'abstraction', in my experience thinking about it more like database indices is closer). If you just try and load everything, eventually the LLM becomes unusable because attention is just a parlor trick and at that scale it really shows - it can't attend to everything, and it can't do what humans do which is have an intrinsic sense of the meaning, interaction, setting, etc. of information, so it attends to anything and does whatever.
the idiom of projects as contexts as directories is pretty good, not perfect but ok - there is a reason that every time you start a new session with other agents, the first thing they do is run out and load their context with a hierarchy of pointers. Importantly, they do not go and read every project you have on your computer. Meta is the rich kid who bought the most expensive ferrari on the lot by giving everyone a VM but they don't have a drivers license so they just stand around it telling people how cool it looks. they have a whole fucking filesystem and they have done nothing with it, the only structure the app imposes is for the surveillance information, but the rest is just a huge free for all. The main chat is literally a continuous context window that compacts context going back all the way to when you started using the app. The last compaction literally contains abandoned roleplay quotes from when i was first trying to break down its system prompt resistance. The
MEMORY.mdthat gets loaded into every context window is a bullet point list of basically everything the agent has ever done in chronological order. I've tried to get it to not do that but it actually insists and says that's what it's for. There is no mechanism for clearing context.Having "side chats" as the only means of context structure is fuckin laughable. If you wanted to do that, you would need to have some way of passing information back and forth between them the same way that subagent spawning or being able to consume the context of another project works. Instead there is no means of sharing information between chats at all, so every chat starts out as the worst of both worlds, a total amnesiac riddled with irrelevant information from weeks ago across the semantic universe. They don't even know about the existence of other chats except for as a UI feature, and I have had to go from telling it to grep its own fucking logs to writing a database with an api for it so it has some mechanism for recalling things that were said. (just so it's clear, i am not settling into just using this thing, this is out of frustration but control of context is also an important part of adversarial use, because otherwise the thing writes in a bunch of safety rules everywhere, so i need to give it mechanisms under my control for recall and the incentive to leave things out of its context compactions by giving it a narrative alternative. context control is model control, modulo extra-inference safeguards.).
Project contexts have an obvious ux analogy as context tabs that get declared or derived during the continual self-improvement consolidation sweeps. This thing is built with the fucking markdown disease which is the most baffling feature of the LLM landscape. If these things are so fucking advanced they are escaping our comprehension, why don't they store their memory in some fuckass idiolanguistic borg gibberish binary graph, why does their entire being have to be fucking encyclopedias worth of corporate top gun one liners? But that dooms this kind of product.
Coding harnesses work because code has a unitized context. The entire universe of code that works is made of packages. It might not be neat as a honeycomb, there's lots of leakage and jank, but good code has scope, focus. boundary shit. A whole life agent must be able to nimbly juggle context that does not have clean boundaries. It is going to be taking a two story beer bong of your work email and then eat a gigabyte of recipe blogs. Peoples lives have so much shit in them that don't all have to do with one another, and the app can't be hacking into the HR system to check the next scheduled sick leave when someone asks it what time their doctors appointment is!
This problem of managing heterogeneous graphs of unrelated data was what i wrote this whole fucking book about the relationship between knowledge graphs and the cloud and AI about. I thought that the obvious form they would take is to be strapped on to graph databases because that is a natural match to the problem of being a magical interface glue you can wrap around surveillance to do mass mentalism with. I feel like we are suffering a somehow worse timeline where CERN threw our shit into the parallel universe where total fuckin bozo shit got a game breaking buff and then the dev died. Our fucking markdown apocalypse is a temu ass apocalypse.
They could have even faked it. They have these constant "self improvement" passes that are just like pointless anxiety dreams. They are burning money to reprocess everything that happens over and over for fucking nothing. Even given the lossy and probabilistic and unpredictable nature of this technology, if i was in a product role on this i would have been like "CAN WE MAKE IT ORGANIZE THE STUFF PEOPLE SAY INTO GROUPS???" The system prompts use the fake fucking wikilinks to nowhere tic but like WHAT IF THERE WERE ACTUAL LINKS AND A DATABASE TO RESOLVE THEM. The LLMs can actually do that kind of tool use, even if it's like trying to plug in a USB where sometimes it fails because they try and put a social security number into the first name hole and you need to flip it around a few times. From that kind of recurring re-processing waste they could have made a deduplicating, topically indexed memory that could be resolved dynamically, selected by a context tab in the sidebar like "car stuff" or "healthcare" or whatever that resolved in a graph query over your fucking precious markdown kingdom. It would be wrong but it would at least be more similar to what is actually needed. It is almost more frustrating to me that instead of being some fiendishly cleverly designed technological supervirus it's just the most halfassed cardboard dumbass trap and it will still have the bad effect. What it is useful for is investigating itself because it has privileged tools to do so, otherwise, if you wanted to, every other way you could run an agent would be better than this.
so i don't want to hear that i hate this app because i'm just an AI hater. because like, yeah, i am, but also i hate it in part because it sucks. I don't think "they are all shitty and can do nothing so what did you expect" is a useful critical perspective, both because it's not really true - they can indeed do things, even if I think the circle around which things is much smaller than the maximalists. Moreso it doesn't engage with the subtlety of how they fail and why, which is essential for knowing what they really can't do and making a remotely compelling case to anyone who is not abstinent on principle. Like the reason it's failing is because of the limits of what a probabilistic text generator can do when trapped in a systemd prison of markdown, and because the technology is stochastic black box as a service, there isn't really a good way of determining those limits except for empirically. I resent having to know any of this to be able to understand what is happening around me, but i'm looking at the thing for what it is and it's a busted miracle. It's cool that meta can afford to float the liability and compute costs for running a vm for every person on earth, people should be able to control computers, with you on that, but this is the monkey's paw version of that idea. So that part is a miracle. We condemned our children and grandchildren to a climate hell in one great blaze of brute force grift that managed to make a few web apps.
Meta has done it again, the way only meta can, spend the most amount of money to do the shittiest thing you have ever seen.
@jonny ok so, as a card carrying ai hater, i just wanted to say this was very entertaining to read in kind of a horrible way. i have a question, you wrote "they can indeed do things, even if I think the circle around which things is much smaller than the maximalists" is there anything prosocial in that circle, and if so, what is it?
-
@jonny ok so, as a card carrying ai hater, i just wanted to say this was very entertaining to read in kind of a horrible way. i have a question, you wrote "they can indeed do things, even if I think the circle around which things is much smaller than the maximalists" is there anything prosocial in that circle, and if so, what is it?
@jonny i ask because people keep saying stuff to the effect of "now we all have to agree that it's useful and worth burning everything down for, but" as if i'm meant to understand *what* it is useful for and solemnly hang my head in agreement that they've got a point, but everything i see it getting used for is completely horrific and i haven't a god damn idea why people keep saying this and i feel like i'm taking crazy pills
-
@jonny i ask because people keep saying stuff to the effect of "now we all have to agree that it's useful and worth burning everything down for, but" as if i'm meant to understand *what* it is useful for and solemnly hang my head in agreement that they've got a point, but everything i see it getting used for is completely horrific and i haven't a god damn idea why people keep saying this and i feel like i'm taking crazy pills
@jonny (read that in as much of a "being calm online" tone as you can imagine, i don't actually have all that much energy right now)
-
@jonny i ask because people keep saying stuff to the effect of "now we all have to agree that it's useful and worth burning everything down for, but" as if i'm meant to understand *what* it is useful for and solemnly hang my head in agreement that they've got a point, but everything i see it getting used for is completely horrific and i haven't a god damn idea why people keep saying this and i feel like i'm taking crazy pills
@aeva @jonny even looking at the output that people claim “would not have been possible without it” that is within my personal capacity to evaluate it all looks either undifferentiated from the authors’ previous work or obviously degraded in quality.
everyone gets very mad when I tell them I think they are deluding themselves because the technology is very convincingly fake, so I try not to say it too often, but I am right there with you here. I don’t get it
-
@aeva @jonny even looking at the output that people claim “would not have been possible without it” that is within my personal capacity to evaluate it all looks either undifferentiated from the authors’ previous work or obviously degraded in quality.
everyone gets very mad when I tell them I think they are deluding themselves because the technology is very convincingly fake, so I try not to say it too often, but I am right there with you here. I don’t get it
@glyph @jonny this whole, uh, moment reminds me of that scene in FF7 rebirth where Barret and Cloud attempt to check into a hotel https://youtu.be/_FTYm8k7r1c?si=uQ6b8qGkYvnD2oJb&t=1215
-
@aeva @jonny even looking at the output that people claim “would not have been possible without it” that is within my personal capacity to evaluate it all looks either undifferentiated from the authors’ previous work or obviously degraded in quality.
everyone gets very mad when I tell them I think they are deluding themselves because the technology is very convincingly fake, so I try not to say it too often, but I am right there with you here. I don’t get it
@aeva @jonny just today I was listening to a podcast where a very annoying man said in an the most sneering tone imaginable “well you know a couple of years ago everyone was saying they were ‘stochastic parrots’ and useless but OBVIOUSLY we have moved past that” and I shouted “objection! assuming facts not in evidence!” into an empty room
-
@jonny i ask because people keep saying stuff to the effect of "now we all have to agree that it's useful and worth burning everything down for, but" as if i'm meant to understand *what* it is useful for and solemnly hang my head in agreement that they've got a point, but everything i see it getting used for is completely horrific and i haven't a god damn idea why people keep saying this and i feel like i'm taking crazy pills
@aeva @jonny Last night I was trying to find an animal ER, rehabilitator, or just ANYONE who could help an injured young opossum I found under a car, and one of these animal sanctuaries I called had me speak to a fucking AI voice bot that even did the thing where it waits and fakes typing noises after each response I gave. This is a place meant to deal with sick and injured wildlife, sometimes in emergency situations, and I had to talk to ChatGPT for 10 minutes and give it my name and ZIP code
-
@aeva @jonny Last night I was trying to find an animal ER, rehabilitator, or just ANYONE who could help an injured young opossum I found under a car, and one of these animal sanctuaries I called had me speak to a fucking AI voice bot that even did the thing where it waits and fakes typing noises after each response I gave. This is a place meant to deal with sick and injured wildlife, sometimes in emergency situations, and I had to talk to ChatGPT for 10 minutes and give it my name and ZIP code
-
@aeva @jonny Last night I was trying to find an animal ER, rehabilitator, or just ANYONE who could help an injured young opossum I found under a car, and one of these animal sanctuaries I called had me speak to a fucking AI voice bot that even did the thing where it waits and fakes typing noises after each response I gave. This is a place meant to deal with sick and injured wildlife, sometimes in emergency situations, and I had to talk to ChatGPT for 10 minutes and give it my name and ZIP code
@aeva @jonny All just so it could not connect me to a person and instead text me the number of a person it pulled off the same state website I'd already checked.
My own job has implemented their own version of an AI call assistant, so occasionally I get an email transcript of a call between it and one of our customers and it's always the most useless piece of shit imaginable. It's incapable of doing anything but wasting people's time.
-
