People get mad when you call LLMs "spicy autocomplete" but my investigations into recreating and implementing small versions of this tech make me think that nick name is very accurate.
-
People get mad when you call LLMs "spicy autocomplete" but my investigations into recreating and implementing small versions of this tech make me think that nick name is very accurate.
Basically, it's a method to predict the next content in a text file. The whole conversation between you and the LLM is one file, and the LLM tries to find the most likely next text based on the training data.
There is something significant here: LLMs were trained on internet forums and social media.
Thus the training data didn't just contain text, but rather text where each passage is tagged and attributed to a particular user.
This aspect of the training data was critical in creating the illusion of talking to another person.
An LLM doesn't just predict the next text. It predicts the next text that might come from another user. You need to hard code this in to make it work well.
Leave it out and there is no conversation.
-
People get mad when you call LLMs "spicy autocomplete" but my investigations into recreating and implementing small versions of this tech make me think that nick name is very accurate.
Basically, it's a method to predict the next content in a text file. The whole conversation between you and the LLM is one file, and the LLM tries to find the most likely next text based on the training data.
There is something significant here: LLMs were trained on internet forums and social media.
@futurebird
it's so similar to the Markov chain generators we used to play with on IRC, just backed with obscene amounts of compute and fancily parsed data instead of a text file and some random spare cycles -
Thus the training data didn't just contain text, but rather text where each passage is tagged and attributed to a particular user.
This aspect of the training data was critical in creating the illusion of talking to another person.
An LLM doesn't just predict the next text. It predicts the next text that might come from another user. You need to hard code this in to make it work well.
Leave it out and there is no conversation.
For example if I give an LLM without user seperation this text:
"It's a lovely day." It might continue with "The sun was shining."
But with user separation it focuses on responses to "it was a lovely day" from other users and the training data might suggest "I agree, it's wonderful weather."
So interaction with an LLM is like posting on a forum, it gives you and average of typical responses with one small change: most LLMs have a strong positivity bias programmed in.
-
People get mad when you call LLMs "spicy autocomplete" but my investigations into recreating and implementing small versions of this tech make me think that nick name is very accurate.
Basically, it's a method to predict the next content in a text file. The whole conversation between you and the LLM is one file, and the LLM tries to find the most likely next text based on the training data.
There is something significant here: LLMs were trained on internet forums and social media.
@futurebird Which is weird, because you'd think "You're absolutely right" wouldn't be anywhere in their lexicon.
-
For example if I give an LLM without user seperation this text:
"It's a lovely day." It might continue with "The sun was shining."
But with user separation it focuses on responses to "it was a lovely day" from other users and the training data might suggest "I agree, it's wonderful weather."
So interaction with an LLM is like posting on a forum, it gives you and average of typical responses with one small change: most LLMs have a strong positivity bias programmed in.
Because let's be real, if you posted "It's a lovely day." on an internet forum you might get a response like "No it's not, noob."
LLMs are heavily weighted to give supportive, and constructive responses.
I wonder what they might be like without these limitations? Without the limitation to make the response from another user they might be much less deceptive.
That they are popular shows that many people just want a nice moderated online community where people treat each other with respect.
-
People get mad when you call LLMs "spicy autocomplete" but my investigations into recreating and implementing small versions of this tech make me think that nick name is very accurate.
Basically, it's a method to predict the next content in a text file. The whole conversation between you and the LLM is one file, and the LLM tries to find the most likely next text based on the training data.
There is something significant here: LLMs were trained on internet forums and social media.
@futurebird this is also literally what actual researchers in machine learning describe. You're absolutely correct in your assessment.
-
The people who said "no calling it spicy autocomplete misses the whole point there is more going on!"
Really made me think that maybe there was more going on, and of course they'd never say. But, it's just the rather clever exploit of using the way that so much of the training data was in the form of posts and responses to make the auto completer feel more like a conversation.
That is the "more" that is going on.
-
@futurebird Which is weird, because you'd think "You're absolutely right" wouldn't be anywhere in their lexicon.
Well isn't that what we all long to hear when we post online? Here is a program that will ALWAYS do that. No trolls, not prickly experts pointing our your spelling errors, no people who are right and trying to tell you why you are wrong with more patience than you deserve.
I do think there is a lesson here. You can get a long way with people just by not being a jerk.
It's one of the reasons I like the fedi. People will say when you are wrong but they are nice about it... mostly.
-
People get mad when you call LLMs "spicy autocomplete" but my investigations into recreating and implementing small versions of this tech make me think that nick name is very accurate.
Basically, it's a method to predict the next content in a text file. The whole conversation between you and the LLM is one file, and the LLM tries to find the most likely next text based on the training data.
There is something significant here: LLMs were trained on internet forums and social media.
@futurebird maybe the “spicy” part is really what’s wrong with it, given the outputs are so bland
-
@futurebird
it's so similar to the Markov chain generators we used to play with on IRC, just backed with obscene amounts of compute and fancily parsed data instead of a text file and some random spare cycles"Big Markov",
"Deep Markov"
or, terrifyingly
"General Purpose Markov" ? -
People get mad when you call LLMs "spicy autocomplete" but my investigations into recreating and implementing small versions of this tech make me think that nick name is very accurate.
Basically, it's a method to predict the next content in a text file. The whole conversation between you and the LLM is one file, and the LLM tries to find the most likely next text based on the training data.
There is something significant here: LLMs were trained on internet forums and social media.
@futurebird @jannem Yup. It’s a weighted random generator auto complete sold to idiots that it will replace the whole workforce and make those idiots bazillions while also taking away peoples possibility to actually purchase hardware they own. It’s an auto complete enslavement device
-
Ian, you had me going for a moment there. I was like "how do they keep finding me? why are they like this all the time???"
-
The wind up with the bamboozling jargon (you can feel these dudes hoping they put in enough tricky sounding word and concepts to make you just give up) was perfect.
"token prediction"
"vectors"
"gradient descent" (OMG)The problem is math jargon is my briar patch and tossing me in there is a big mistake.
-
Ian, you had me going for a moment there. I was like "how do they keep finding me? why are they like this all the time???"
@futurebird@sauropods.win I honestly think (unpopular opinion here) that most of the cost of LLM-based AI thus far is in ‘training’. Not training as in running the phenomenal amount of harvested stolen text and image input through tokenisation processes and reward giving through weight assignment and vector assessment, using more GPUs than exist on Earth, but rather, lots and lots and lots of money paying humans to fake it all and build in patches – patch after patch on top of patch of corrective behaviour, encoded themselves as vector weights. The training had nothing much to do with running it all through GPUs, I believe that probably took an embarassing but totally affordable amount of time and energy. I believe (with no visible means of factual reference to cite) that most of the expenditure of these capital-burning companies was ‘training’ by paying humans and then encoding their resulting guidance. Paying workers.
-
[Not arguging that these models are 'thinking', even if it might sound like that.]
I think the "explain how you arrived at that conclusion" that was all the rage is very interesting for two reasons:
The modell is generating more text. It's not like it is showing you a walk through its model and the random numbers it pulled. So it is basically generating an explanation that is plausible given what the said before.
I think this is often also a behavior with humans. My opinion about a topic might be a gut feeling, but when questioned I start thinking about it, trying to find arguments. Often ones I didn't already have when I stated my opinion.
The first thing could make sense to ask if amodel are trained to change their position given new information. So they could "correct" a bad roll of the dice.
Of course, a user might think that the model "really thought about this", which is obviously not the case. -
The wind up with the bamboozling jargon (you can feel these dudes hoping they put in enough tricky sounding word and concepts to make you just give up) was perfect.
"token prediction"
"vectors"
"gradient descent" (OMG)The problem is math jargon is my briar patch and tossing me in there is a big mistake.
@futurebird@sauropods.win I wasn’t making it up though - the way it works is by tokenising language (not into words but into fragments of words), then assigning the word-derived tokens to vectors (word2vec - it exists), then these vectors are winnowed into the likely winners by gradient descent to find the lowest error (and not get trapped by just falling downhill down the nearest valley) and so on.
-
Well isn't that what we all long to hear when we post online? Here is a program that will ALWAYS do that. No trolls, not prickly experts pointing our your spelling errors, no people who are right and trying to tell you why you are wrong with more patience than you deserve.
I do think there is a lesson here. You can get a long way with people just by not being a jerk.
It's one of the reasons I like the fedi. People will say when you are wrong but they are nice about it... mostly.
"It's one of the reasons I like the fedi. People will say when you are wrong but they are nice about it... mostly"
I find this unfortunately being less and less so as more people discover it. Lately I've seen way more flaming and obnoxious argumentativeness than ever previously. Sigh.
-
"It's one of the reasons I like the fedi. People will say when you are wrong but they are nice about it... mostly"
I find this unfortunately being less and less so as more people discover it. Lately I've seen way more flaming and obnoxious argumentativeness than ever previously. Sigh.
That sucks. Was it someone you knew acting differently or new people showing up? I hardly ever see any real drama so I'm kind of curious in an shallow gossip driven way what was going down...
-
That sucks. Was it someone you knew acting differently or new people showing up? I hardly ever see any real drama so I'm kind of curious in an shallow gossip driven way what was going down...
@futurebird @woe2you I stick mostly to animal photography on my timeline, which still seems friendly and unaffected. But when looking at trending posts, so seeing things I wouldn't normally see, there tends to be more arguing. Unsurprisingly, it's usually political posts, which are always going to raise emotions, but people used to at least argue constructively. Now it seems to be a lot of yelling and swearing. Not everywhere, but more often.
-
"Big Markov",
"Deep Markov"
or, terrifyingly
"General Purpose Markov" ?
