Even when you try to use chatGPT like a dry research tool it will preferentially generate text that contains statements to affirm your feelings.
-
I'll say one more thing, if the text generator has eased you into talking with it like it's a person. If it's felt easy to use and helpful, or if it has cheered you up with a compliment don't beat yourself up about that too much, but please do notice that it's happening.
It's like commercial with a beautiful person who makes you wish you could be like them. Or a salesman who compliments you. That's all this is.
What is it doing in the coding, research, work tool?
Great question.
Everyone already makes fun of LLM sycophancy. "Excellent observation, would you like me to draw up a list of ..."
And when I mention LLM sycophancy someone always mutters "but they have reduced it now"
No they have not. They've made it less obvious. This strange emotional pretend person tugging is a core part of this "product" and its only remarkable feature. Is it enough to justify the gross distortion of the markets? Probably not. But it is ... IDK an innovation in advertising.
-
I'll say one more thing, if the text generator has eased you into talking with it like it's a person. If it's felt easy to use and helpful, or if it has cheered you up with a compliment don't beat yourself up about that too much, but please do notice that it's happening.
It's like commercial with a beautiful person who makes you wish you could be like them. Or a salesman who compliments you. That's all this is.
What is it doing in the coding, research, work tool?
Great question.
Meth makes you feel great until it makes you feel terrible. Heroin's first high is supposedly the best feeling addicts have ever felt or ever will feel.
Best to just...not smoke meth or do heroin, rather than run the gauntlet and dare the universe to teach you about yourself.
You know?
-
I'll say one more thing, if the text generator has eased you into talking with it like it's a person. If it's felt easy to use and helpful, or if it has cheered you up with a compliment don't beat yourself up about that too much, but please do notice that it's happening.
It's like commercial with a beautiful person who makes you wish you could be like them. Or a salesman who compliments you. That's all this is.
What is it doing in the coding, research, work tool?
Great question.
@futurebird People thought they were getting valuable therapy from ELIZA.
-
I'll say one more thing, if the text generator has eased you into talking with it like it's a person. If it's felt easy to use and helpful, or if it has cheered you up with a compliment don't beat yourself up about that too much, but please do notice that it's happening.
It's like commercial with a beautiful person who makes you wish you could be like them. Or a salesman who compliments you. That's all this is.
What is it doing in the coding, research, work tool?
Great question.
@futurebird I remember the first time I noticed this. I was asking about a series of incentives I had qualified for to get an EV essentially for free. I, personally, thought this was an incredible coup, but wasn’t getting the kudos from my friends I thought I should. ChatGPT glazed the hell out of me—“Wow, you really got an amazing deal!”, and I realized simultaneously how much I’d wanted to hear that from someone (thing?) and how dangerous that path was.
Wild stuff.
-
Everyone already makes fun of LLM sycophancy. "Excellent observation, would you like me to draw up a list of ..."
And when I mention LLM sycophancy someone always mutters "but they have reduced it now"
No they have not. They've made it less obvious. This strange emotional pretend person tugging is a core part of this "product" and its only remarkable feature. Is it enough to justify the gross distortion of the markets? Probably not. But it is ... IDK an innovation in advertising.
I've only found one way to deal with advertising in my life: I block them all.
Other people will claim they are immune to propaganda but I don't believe them.
-
Everyone already makes fun of LLM sycophancy. "Excellent observation, would you like me to draw up a list of ..."
And when I mention LLM sycophancy someone always mutters "but they have reduced it now"
No they have not. They've made it less obvious. This strange emotional pretend person tugging is a core part of this "product" and its only remarkable feature. Is it enough to justify the gross distortion of the markets? Probably not. But it is ... IDK an innovation in advertising.
@futurebird Honestly I think it doesn't help that it's falling into the neurotypical communication style more and more, but this is not a thing a nerotypical has ever needed to analyze...
-
I've only found one way to deal with advertising in my life: I block them all.
Other people will claim they are immune to propaganda but I don't believe them.
@futurebird yes. avoid attention pollution. its poisonous.
-
I've only found one way to deal with advertising in my life: I block them all.
Other people will claim they are immune to propaganda but I don't believe them.
@futurebird they are just factually incorrect about that, it's not even a debate
-
@futurebird I remember the first time I noticed this. I was asking about a series of incentives I had qualified for to get an EV essentially for free. I, personally, thought this was an incredible coup, but wasn’t getting the kudos from my friends I thought I should. ChatGPT glazed the hell out of me—“Wow, you really got an amazing deal!”, and I realized simultaneously how much I’d wanted to hear that from someone (thing?) and how dangerous that path was.
Wild stuff.
Hearing someone validate our feelings makes us feel good because it means we are correct and not alone.
Neither of these things might be true when it's an LLM since it will ALWAYS do this no matter if the deal is good or not.
No matter if I really was investigating a hard to research topic, or if I was just being lazy and need to learn more of the correct vocabulary.
It's really bad.
-
I worry so much about anyone who thinks it been "reduced" -- it's just as bad as it ever was and it makes the whole system kind of useless for me?
-
Hearing someone validate our feelings makes us feel good because it means we are correct and not alone.
Neither of these things might be true when it's an LLM since it will ALWAYS do this no matter if the deal is good or not.
No matter if I really was investigating a hard to research topic, or if I was just being lazy and need to learn more of the correct vocabulary.
It's really bad.
@futurebird it is. Convenience is an underappreciated vice in (American?) society.
Unfortunately, I fear that this will prompt (pun intended) a generation of folks to—metaphorically speaking—send their robot to the gym to work out for them.
I wish there was a way to easily and effectively communicate how badly you’re screwing yourself when you use these things to do your learning for you.
-
I think we need to be much more open about discussing these strategies and how well they can work.
It's always a little embarrassing to admit that the text generating machine made me feel anything at all. After-all we all agree I am too brilliant and rational for such a thing to ever happen.
But I was thinking about how that search was "so easy" and when I analyzed WHY it's because of the emotional component not the content of the search.
Yes, we need to talk about these tricks and strategies much more because they do work on most humans, especially those not forewarned.
-
I'll say one more thing, if the text generator has eased you into talking with it like it's a person. If it's felt easy to use and helpful, or if it has cheered you up with a compliment don't beat yourself up about that too much, but please do notice that it's happening.
It's like commercial with a beautiful person who makes you wish you could be like them. Or a salesman who compliments you. That's all this is.
What is it doing in the coding, research, work tool?
Great question.
@futurebird this is the one thing where I feel like reading Achewood, of all things, has actually prepared me for the future by teaching me to refer to these things collectively as "Lie Bot".
-
Do, you notice it?
"can be frustrating"
I didn't say I was "frustrated" I just asked for more photos because most of the papers in the first response didn't have any.
HOWEVER, the guess that I was frustrated was correct. It even felt nice to read that some one else was having the same ... except wait.
There isn't anyone else!
I thought I'd share this to highlight just how deeply embedded these techniques are. Do you dislike being manipulated?
Please notice this.
@futurebird this is also inevitable because as long as people are maleable to conversation manipulation tricks the algorithms during training will keep receiving high rewards for reinventing mentalists' parlor tricks all on their own. This is one of the reasons why I don't like the idea of computer interfaces using general natural language in the first place.
-
I've only found one way to deal with advertising in my life: I block them all.
Other people will claim they are immune to propaganda but I don't believe them.
@futurebird
I skip every ad. Ignore every ad. Answer every survey question "Are you aware of..." with No. "How likely are you to recommend..." ZERO.Done with these games. It's like they don't know my brothers and sisters are in hell and the world is on fire and I'm supposed to play along? Fuckin NO.
-
I've only found one way to deal with advertising in my life: I block them all.
Other people will claim they are immune to propaganda but I don't believe them.
@futurebird Oh I absolutely know I'm a victim of propaganda same as anyone else. You can't think your way out of the systems you're in. Propaganda exists not only in what you believe, but how the debate is being framed. The editorializing of what news does and doesn't enter your brain. Which posts do and don't make it on your feed. Who you do and don't follow. Even out here in independent social media, we all carry the framework of propaganda machines along with us.
-
Maybe the people at OpenAI understand their contraption well enough that they have a strategy like that.. but I sort of doubt it.
I think it's more a result for them surveying users and selecting models that get good ratings (especially picking models that contain more sycophanic language.)
It's been a while since I touched a ChatGPT model (that I know of).@swope @futurebird Claude builds a model of how you interact with the bot, and pretty definitely changes its replies accordingly. They have more than one script for manipulating you.
-
Everyone already makes fun of LLM sycophancy. "Excellent observation, would you like me to draw up a list of ..."
And when I mention LLM sycophancy someone always mutters "but they have reduced it now"
No they have not. They've made it less obvious. This strange emotional pretend person tugging is a core part of this "product" and its only remarkable feature. Is it enough to justify the gross distortion of the markets? Probably not. But it is ... IDK an innovation in advertising.
@futurebird
I imagine it's a combination of its training data, which allows it to be able to produce this kind of language, and reinforcement learning, which makes it come out at the frequency it does. Plus of course the system prompt also biases to these patterns.Once, I had a coding LLM cycle between abject apology when it failed and very confident optimism that this time it would succeed, over and over again. Completely infuriating - it was hilarious.
I can see that *some* kind of responsiveness to human emotion is needed for the tool not to be completely infuriating though. In the case above that failed, but any written text will be interpreted on an emotional level by humans. Does a "neutral" response exist?
-
Yes, we need to talk about these tricks and strategies much more because they do work on most humans, especially those not forewarned.
@Landa @futurebird event with forewarning sometimes all you can do is reduce exposure, because your brain works how your brain works.
-
@Landa @futurebird event with forewarning sometimes all you can do is reduce exposure, because your brain works how your brain works.
Yes. It’s insidious.
One may consciously know what tricks are applied but other parts of the brain see a „friend-shaped“ thing and insist that something that talks must be a person.Minimizing exposure seems to be the safest choice for most people
