This blogpost makes an astoundingly good case about LLMs I hadn't considered before.
-
This blogpost makes an astoundingly good case about LLMs I hadn't considered before. The collapse of public forums (like Stack Overflow) for programming answers coincides directly with the rise of programmers asking for answers from chatbots *directly*. Those debugging sessions become part of a training set that now *only private LLM corporations have access to*. This is something that "open models" seemingly can't easily fight. https://michiel.buddingh.eu/enclosure-feedback-loop
@cwebber as in many other fields, we have to have real communities who care about stuff.
-
@cwebber I've been saying this for a while. Bubble or not, our profession (and/or vocation, if you prefer) is screwed.
@datarama Possibly, though I worry less about professions/vocations than I do about user empowerment. I have long assumed that some day programmer salaries would be unsustainable.
Of course the irony is that many people are shilling LLM services as being empowerment systems. I see them as the opposite. Open, community developed LLMs could be, but LLM-as-a-service corporations are definitively not.
-
@datarama Possibly, though I worry less about professions/vocations than I do about user empowerment. I have long assumed that some day programmer salaries would be unsustainable.
Of course the irony is that many people are shilling LLM services as being empowerment systems. I see them as the opposite. Open, community developed LLMs could be, but LLM-as-a-service corporations are definitively not.
@cwebber By vocation, I also mean "people who like to write software".
If I lost my job but still had that, I'm sure I could become a happy store clerk or train driver who hacked on community software in my free time. But in AI Hell, we can't even have that. My option is to become a miserable store clerk or train driver (until that too is automated away) who consumes AI-generated slop forever. And that is what is coming for all of us - current-day programmers are just going to get there first.
(Incidentally, I make less than a third of what people on the internet tell me American software developers with my level of experience do - but I'm no more or less screwed than they are.)
-
@datarama Possibly, though I worry less about professions/vocations than I do about user empowerment. I have long assumed that some day programmer salaries would be unsustainable.
Of course the irony is that many people are shilling LLM services as being empowerment systems. I see them as the opposite. Open, community developed LLMs could be, but LLM-as-a-service corporations are definitively not.
@cwebber And the problem is, LLM development is *extremely* capital-intensive. Unless you have a "community" of billionaires, it's going to be very hard to make anything that can compete with the hyperscalers.
-
@cwebber By vocation, I also mean "people who like to write software".
If I lost my job but still had that, I'm sure I could become a happy store clerk or train driver who hacked on community software in my free time. But in AI Hell, we can't even have that. My option is to become a miserable store clerk or train driver (until that too is automated away) who consumes AI-generated slop forever. And that is what is coming for all of us - current-day programmers are just going to get there first.
(Incidentally, I make less than a third of what people on the internet tell me American software developers with my level of experience do - but I'm no more or less screwed than they are.)
@datarama @cwebber I can attest that it's still possible to hack on free software in your spare time if you lose the tech job, but you get a heck of a lot less free time to do it in. And a heck of a lot less energy to do it with. All against a billionaire-induced media backdrop of your primary interest now being irrelevant, which is demoralizing.
But if you can find the time and maintain the energy, there is still a community even more stubborn than in the "GPL is a cancer" days.
-
@datarama @cwebber I can attest that it's still possible to hack on free software in your spare time if you lose the tech job, but you get a heck of a lot less free time to do it in. And a heck of a lot less energy to do it with. All against a billionaire-induced media backdrop of your primary interest now being irrelevant, which is demoralizing.
But if you can find the time and maintain the energy, there is still a community even more stubborn than in the "GPL is a cancer" days.
@randomgeek @cwebber It's *possible*, of course, but it all feels rather pointless now.
And everything you make and share freely is appropriated to improve the Immiseration Machine.
-
@cwebber but also, as uninviting as the stack overflow culture may have been, the moderators were there to try to get people to ask better questions. I doubt llms will handle things like x/y problem issues, so to me it seems things will get worse for people able/willing to pay as well.
@martijn @cwebber IMHO stackoverflow may have been toxic, but it was a sort of forum with low friction access (easy to search, easy to ask, easy to reply) where you interacted WITH PEOPLE.
People is key. I remember names from the linux-kernel list in the mid-90s - I joined Mastodon in 2022 and found that same people here.
Whatever site or forum or network or anything we build, I want to read from people, not bots.
-
@randomgeek @cwebber It's *possible*, of course, but it all feels rather pointless now.
And everything you make and share freely is appropriated to improve the Immiseration Machine.
-
@randomgeek @cwebber I'm also autistic. (though I'm Danish, so the *least* famously crazy kind of Scandinavian.
)In the beginning of all this, I thought and felt much the same. Now I just feel drained and defeated.
Because yes, the struggle itself is enough to fill a human heart and we must imagine Sisyphus happy. But it sucks to be Sisyphus when someone put up a ski lift next to him.
-
This blogpost makes an astoundingly good case about LLMs I hadn't considered before. The collapse of public forums (like Stack Overflow) for programming answers coincides directly with the rise of programmers asking for answers from chatbots *directly*. Those debugging sessions become part of a training set that now *only private LLM corporations have access to*. This is something that "open models" seemingly can't easily fight. https://michiel.buddingh.eu/enclosure-feedback-loop
@cwebber I think there is a flaw with the theory that big AI can use this shift from forum to chatbot to train new models. The thing that makes Stack Overflow valuable is not the question but having an expert(s) provide an answer, and a mechanism for others to add weight to it being correct.
Interactions with LLMs really don't have the same feedback loop. They collect the questions from the users, but there is no expert to provide the answer to train from. I suppose there's some training data there, but not nearly as direct as what was originally scraped from SO.
I suspect training future models is going to be much more challenging.
-
This blogpost makes an astoundingly good case about LLMs I hadn't considered before. The collapse of public forums (like Stack Overflow) for programming answers coincides directly with the rise of programmers asking for answers from chatbots *directly*. Those debugging sessions become part of a training set that now *only private LLM corporations have access to*. This is something that "open models" seemingly can't easily fight. https://michiel.buddingh.eu/enclosure-feedback-loop
@cwebber yet another externality for the bot lickers to ignore when they say "ethical and environmental issue aside..." and praise the occasionally useful slop that the stochastic slotmachine gives them as they burn billions of tokens in gas town.
-
This blogpost makes an astoundingly good case about LLMs I hadn't considered before. The collapse of public forums (like Stack Overflow) for programming answers coincides directly with the rise of programmers asking for answers from chatbots *directly*. Those debugging sessions become part of a training set that now *only private LLM corporations have access to*. This is something that "open models" seemingly can't easily fight. https://michiel.buddingh.eu/enclosure-feedback-loop
@cwebber I think this is clearly right about enclosure, but wrong about there being a positive side of the loop that helps make LLMs better. When people ask an LLM for help, it just regurgitates old answers, it can't generate new ones. This generates training data about what questions people have, but does not generate training data about solutions except in rare cases where the user figures out their issue themselves and chats about the solution with the agent. The human experts answering the questions on SO part of entirely missing from the LLM interaction, unless the solution was *already* in the training data.
-
This blogpost makes an astoundingly good case about LLMs I hadn't considered before. The collapse of public forums (like Stack Overflow) for programming answers coincides directly with the rise of programmers asking for answers from chatbots *directly*. Those debugging sessions become part of a training set that now *only private LLM corporations have access to*. This is something that "open models" seemingly can't easily fight. https://michiel.buddingh.eu/enclosure-feedback-loop
@cwebber This goes much, much wider than programming and LLMs.
In general, the open source world looks with disdain at all kinds of automated feedback collection mechanisms, which the Silicon Valley Venture Capital tech ecosystem has wholeheartedly embraced. OSS is still stuck in the 1990s mindset of "if there's a problem, somebody will report this to us", and That... just isn't true.
What we're stuck with is OSS solutions with inferrior user experiences which nobody wants to use, instead of a compromise where OSS software collects more data than some people would have liked, but that software actually has some users and makes a difference in the world.
To be fair, there are some good arguments against this (it's much easier to protect user privacy if the only contributors to your code are employees with background checks), but that doesn't make this less of a problem.
-
@mahadevank @cwebber Forget trying to explain that. The "experts" at Davos laid it out for everyone. Yet, somehow they're still optimistic that one entity dominating all others, essentially destroying competition, will bring forth a world of opportunities. It's an all out war, and anyone that doesn't have the resources to insert XYZ's brain into their stack, is just a foot soldier for those that do.
@cmthiede @cwebber ah but the world and nature don't work this way - I mean, we arrived at these systems after realizing that the tyrannical and control-driven systems of yesteryears were never stable.
The Imperials of Davos may think this way, but that's never how it comes to pass. Let them enjoy their rather small window of opportunity while it lasts.
-
@cmthiede @cwebber ah but the world and nature don't work this way - I mean, we arrived at these systems after realizing that the tyrannical and control-driven systems of yesteryears were never stable.
The Imperials of Davos may think this way, but that's never how it comes to pass. Let them enjoy their rather small window of opportunity while it lasts.
@mahadevank @cwebber I suppose you're right. I've visited spectacular pyramids of civilizations past, these "hyperscalers" are just another chapter in the long book of history.
-
@cwebber This goes much, much wider than programming and LLMs.
In general, the open source world looks with disdain at all kinds of automated feedback collection mechanisms, which the Silicon Valley Venture Capital tech ecosystem has wholeheartedly embraced. OSS is still stuck in the 1990s mindset of "if there's a problem, somebody will report this to us", and That... just isn't true.
What we're stuck with is OSS solutions with inferrior user experiences which nobody wants to use, instead of a compromise where OSS software collects more data than some people would have liked, but that software actually has some users and makes a difference in the world.
To be fair, there are some good arguments against this (it's much easier to protect user privacy if the only contributors to your code are employees with background checks), but that doesn't make this less of a problem.
-
@cwebber I think this is clearly right about enclosure, but wrong about there being a positive side of the loop that helps make LLMs better. When people ask an LLM for help, it just regurgitates old answers, it can't generate new ones. This generates training data about what questions people have, but does not generate training data about solutions except in rare cases where the user figures out their issue themselves and chats about the solution with the agent. The human experts answering the questions on SO part of entirely missing from the LLM interaction, unless the solution was *already* in the training data.
@tiotasram @cwebber The claim seems to be that frustrated users who do know the right answers argue with the chatbots, giving them new training material. If true this suggests a fun attack...
-
@cwebber I think there is a flaw with the theory that big AI can use this shift from forum to chatbot to train new models. The thing that makes Stack Overflow valuable is not the question but having an expert(s) provide an answer, and a mechanism for others to add weight to it being correct.
Interactions with LLMs really don't have the same feedback loop. They collect the questions from the users, but there is no expert to provide the answer to train from. I suppose there's some training data there, but not nearly as direct as what was originally scraped from SO.
I suspect training future models is going to be much more challenging.
@matsuzine @cwebber it’s like a snake eating its own tail, eventually there is nothing left to eat.
-
@randomgeek @datarama @cwebber New tourist slogan for Finland - We're all Autistic Here!
-
This blogpost makes an astoundingly good case about LLMs I hadn't considered before. The collapse of public forums (like Stack Overflow) for programming answers coincides directly with the rise of programmers asking for answers from chatbots *directly*. Those debugging sessions become part of a training set that now *only private LLM corporations have access to*. This is something that "open models" seemingly can't easily fight. https://michiel.buddingh.eu/enclosure-feedback-loop
@cwebber I wonder what will be the solution to that.
for a while now I've been thinking of a return to a more "cathedral" way of doing things, specifically as a reaction against LLMs. software developed by small teams of trusted devs in private repos, communication structure that resembles BBSes and geocities webrings more than web 2.0 social media. obscure blogs in gemini capsules, any shadowed enough corner to escape the tendrils of the USA miltech complex. I don't see any solutions to slop that don't involve guarded human curation and networks of trust.
for documentation, I remember how it was before StackExchange; I used to learn everything from /usr/share/doc/HOWTO, from info libc, from carefully written technical documentation that was distributed offline and that you could read cover to cover. the occasional o'reilly book for more complex topics. maybe it's too much to expect a return to this mode of knowledge sharing, but it would make sense against the LLMs, I think—in the same way that I now have to care whether my supposedly open-source software accepts slop code or not, if I have decided I trust the developers of a given piece of software, presumably I also trust the documentation they provide.
it would be an even better result if the tsunami of slop led people to an increased appreciation of the work provided by tech writers, translators, designers and other non-programming labour, though even I am not that optimistic to hope for that. sensible thought it might be.
a return to a /usr/share/doc/HOWTO approach of learning would also entail a certain shift in how we deal with the code itself, prizing human intelligibility and right-sizedness over productivity, quantity-over-quality, increased levels of abstraction/virtualisation/frameworkification. you can't really know that the code isn't LLM trash unless the code is intelligible in the first place, and for that you want it to be the exact oppose of vibe-slop: concise, logical, readable, maintainable over all else. stability and tried-and-tested lines prioritised over chasing trendy features. an attitude less like linux, more like netbsd.
