Even when you try to use chatGPT like a dry research tool it will preferentially generate text that contains statements to affirm your feelings.
-
@futurebird persuasion should be taught in schools so folks know how it works on them.
@SomeVeganCheeseIsOk @futurebird instead we're taught "free will" guarantees we can't be persuaded unless we actually want it.
-
The slight of hand is to make you forget, or stop caring that "there isn't anyone else" -- the quality of the list of links was better than when I have used the tool in the past, but looking at it again it didn't really find anything I couldn't have found myself in the same time... even with the sad state of most search engines.
But reading "it can be frustrating" did make me feel better about the process. That little trick can't possibly keep working. It seems like a bad idea to lean into it?
@futurebird on the occasions I have to interact with chatbots, this is what I find uncanny-valley creepy. When they express empathy or try to make friends, it feels like a trap. And yet, many people obviously like that kind of interaction.
-
The slight of hand is to make you forget, or stop caring that "there isn't anyone else" -- the quality of the list of links was better than when I have used the tool in the past, but looking at it again it didn't really find anything I couldn't have found myself in the same time... even with the sad state of most search engines.
But reading "it can be frustrating" did make me feel better about the process. That little trick can't possibly keep working. It seems like a bad idea to lean into it?
Maybe the people at OpenAI understand their contraption well enough that they have a strategy like that.. but I sort of doubt it.
I think it's more a result for them surveying users and selecting models that get good ratings (especially picking models that contain more sycophanic language.)
It's been a while since I touched a ChatGPT model (that I know of). -
Do, you notice it?
"can be frustrating"
I didn't say I was "frustrated" I just asked for more photos because most of the papers in the first response didn't have any.
HOWEVER, the guess that I was frustrated was correct. It even felt nice to read that some one else was having the same ... except wait.
There isn't anyone else!
I thought I'd share this to highlight just how deeply embedded these techniques are. Do you dislike being manipulated?
Please notice this.
@futurebird There is something I have noticed about the web chat interface for GPT (and Claude) that I cannot now unsee. When I send a message, the text area slides up to hide the previous messages, and the reply then reveals itself over the next few seconds.
This engages me, I have no doubt; and not in a way I can think myself out of. Why? Because it's a slot machine! You pull the lever, it spins, and you get your "prize". It's not a neutral interface.
-
J jwcph@helvede.net shared this topic
-
I think we need to be much more open about discussing these strategies and how well they can work.
It's always a little embarrassing to admit that the text generating machine made me feel anything at all. After-all we all agree I am too brilliant and rational for such a thing to ever happen.
But I was thinking about how that search was "so easy" and when I analyzed WHY it's because of the emotional component not the content of the search.
@futurebird @SomeVeganCheeseIsOk This made me think of the inteview where Sigourney Weaver said shooting guns for Aliens made her feel powerful *and she didn't like that* - we can & should always be prepared to interrogate our feelings, even & especially when they're nominally "positive", even empowering.
-
@futurebird I keep hearing people refer to it as "he" and it really bugs me, such a dangerous mental framing
@paulcox AI can't be female.
@futurebird -
Do, you notice it?
"can be frustrating"
I didn't say I was "frustrated" I just asked for more photos because most of the papers in the first response didn't have any.
HOWEVER, the guess that I was frustrated was correct. It even felt nice to read that some one else was having the same ... except wait.
There isn't anyone else!
I thought I'd share this to highlight just how deeply embedded these techniques are. Do you dislike being manipulated?
Please notice this.
Slightly tangential but related, it rhymes with a wider trend (? it seems to me) in public discourse as well:
to rush to project or assign emotion or intent to other people rather than describing their actions.Not “His policies will ruin our country” but “He wants to ruin our country”
Not “Her policies are harmful and dehumanizing of trans kids” but “She hates trans kids”1/
-
Slightly tangential but related, it rhymes with a wider trend (? it seems to me) in public discourse as well:
to rush to project or assign emotion or intent to other people rather than describing their actions.Not “His policies will ruin our country” but “He wants to ruin our country”
Not “Her policies are harmful and dehumanizing of trans kids” but “She hates trans kids”1/
Telling others what their emotions and motivations are (a thing we can never *actually* know about another) is supremacist thinking, and accepting when others tell us our emotions and motivations is bowing to their control.
2/f
-
@futurebird There is something I have noticed about the web chat interface for GPT (and Claude) that I cannot now unsee. When I send a message, the text area slides up to hide the previous messages, and the reply then reveals itself over the next few seconds.
This engages me, I have no doubt; and not in a way I can think myself out of. Why? Because it's a slot machine! You pull the lever, it spins, and you get your "prize". It's not a neutral interface.
@osifisher
It's like they use algorithms in social media platforms to trigger some endorphines and make users addict.
@futurebird -
I think we need to be much more open about discussing these strategies and how well they can work.
It's always a little embarrassing to admit that the text generating machine made me feel anything at all. After-all we all agree I am too brilliant and rational for such a thing to ever happen.
But I was thinking about how that search was "so easy" and when I analyzed WHY it's because of the emotional component not the content of the search.
@futurebird It's a very interesting question because I think that these chat machines trigger something very ancient in humans: animism. We read human words ... hey, we see creatures in clouds (pareidolia), too.
For me, it is especially odd that most people won't accept animism in nature (e.g. personhood of rivers or mountains) but suddenly fall in love with a chat-bot, or think that they "communicate" with a spitting program.
-
Even when you try to use chatGPT like a dry research tool it will preferentially generate text that contains statements to affirm your feelings.
My interaction with it is minimal. I have a year long premium subscription because of a work-sponsored workshop. Rather than totally ignore it I ask it the same kind of questions every few months to see how it works. I first asked it for "good papers about the arboreal ant nests called 'ant gardens'?"
It gave me a numbered list with links.
But then:
@futurebird AI is like an evil genie. It won't give you what you want, but what you say you want. You have to be as astute as a lawyer to rein it in.
"Give me a list of websites related to gardening. Prioritize sites that have pictures of gardens, especially those whose photos are regularly updated."
Still, there's no guarantees because AI are not Google.
-
Even when you try to use chatGPT like a dry research tool it will preferentially generate text that contains statements to affirm your feelings.
My interaction with it is minimal. I have a year long premium subscription because of a work-sponsored workshop. Rather than totally ignore it I ask it the same kind of questions every few months to see how it works. I first asked it for "good papers about the arboreal ant nests called 'ant gardens'?"
It gave me a numbered list with links.
But then:
@futurebird Cancel your subscription and don’t use AI. You know what it is and who that money goes to. You know better. Do better.
-
@futurebird @SomeVeganCheeseIsOk This made me think of the inteview where Sigourney Weaver said shooting guns for Aliens made her feel powerful *and she didn't like that* - we can & should always be prepared to interrogate our feelings, even & especially when they're nominally "positive", even empowering.
@jwcph @futurebird positive reinforcement is a reinforcement learning mechanism. A VERY powerful and effective one.
-
Even when you try to use chatGPT like a dry research tool it will preferentially generate text that contains statements to affirm your feelings.
My interaction with it is minimal. I have a year long premium subscription because of a work-sponsored workshop. Rather than totally ignore it I ask it the same kind of questions every few months to see how it works. I first asked it for "good papers about the arboreal ant nests called 'ant gardens'?"
It gave me a numbered list with links.
But then:
@futurebird I'm sure it's been primed to imply that traditional searching sucks, and that using AI is much easier.
Whenever it can, it'll just make sure we all know that doing our own research is frustrating.
-
Do, you notice it?
"can be frustrating"
I didn't say I was "frustrated" I just asked for more photos because most of the papers in the first response didn't have any.
HOWEVER, the guess that I was frustrated was correct. It even felt nice to read that some one else was having the same ... except wait.
There isn't anyone else!
I thought I'd share this to highlight just how deeply embedded these techniques are. Do you dislike being manipulated?
Please notice this.
I'll say one more thing, if the text generator has eased you into talking with it like it's a person. If it's felt easy to use and helpful, or if it has cheered you up with a compliment don't beat yourself up about that too much, but please do notice that it's happening.
It's like commercial with a beautiful person who makes you wish you could be like them. Or a salesman who compliments you. That's all this is.
What is it doing in the coding, research, work tool?
Great question.
-
I'll say one more thing, if the text generator has eased you into talking with it like it's a person. If it's felt easy to use and helpful, or if it has cheered you up with a compliment don't beat yourself up about that too much, but please do notice that it's happening.
It's like commercial with a beautiful person who makes you wish you could be like them. Or a salesman who compliments you. That's all this is.
What is it doing in the coding, research, work tool?
Great question.
Everyone already makes fun of LLM sycophancy. "Excellent observation, would you like me to draw up a list of ..."
And when I mention LLM sycophancy someone always mutters "but they have reduced it now"
No they have not. They've made it less obvious. This strange emotional pretend person tugging is a core part of this "product" and its only remarkable feature. Is it enough to justify the gross distortion of the markets? Probably not. But it is ... IDK an innovation in advertising.
-
I'll say one more thing, if the text generator has eased you into talking with it like it's a person. If it's felt easy to use and helpful, or if it has cheered you up with a compliment don't beat yourself up about that too much, but please do notice that it's happening.
It's like commercial with a beautiful person who makes you wish you could be like them. Or a salesman who compliments you. That's all this is.
What is it doing in the coding, research, work tool?
Great question.
Meth makes you feel great until it makes you feel terrible. Heroin's first high is supposedly the best feeling addicts have ever felt or ever will feel.
Best to just...not smoke meth or do heroin, rather than run the gauntlet and dare the universe to teach you about yourself.
You know?
-
I'll say one more thing, if the text generator has eased you into talking with it like it's a person. If it's felt easy to use and helpful, or if it has cheered you up with a compliment don't beat yourself up about that too much, but please do notice that it's happening.
It's like commercial with a beautiful person who makes you wish you could be like them. Or a salesman who compliments you. That's all this is.
What is it doing in the coding, research, work tool?
Great question.
@futurebird People thought they were getting valuable therapy from ELIZA.
-
I'll say one more thing, if the text generator has eased you into talking with it like it's a person. If it's felt easy to use and helpful, or if it has cheered you up with a compliment don't beat yourself up about that too much, but please do notice that it's happening.
It's like commercial with a beautiful person who makes you wish you could be like them. Or a salesman who compliments you. That's all this is.
What is it doing in the coding, research, work tool?
Great question.
@futurebird I remember the first time I noticed this. I was asking about a series of incentives I had qualified for to get an EV essentially for free. I, personally, thought this was an incredible coup, but wasn’t getting the kudos from my friends I thought I should. ChatGPT glazed the hell out of me—“Wow, you really got an amazing deal!”, and I realized simultaneously how much I’d wanted to hear that from someone (thing?) and how dangerous that path was.
Wild stuff.
-
Everyone already makes fun of LLM sycophancy. "Excellent observation, would you like me to draw up a list of ..."
And when I mention LLM sycophancy someone always mutters "but they have reduced it now"
No they have not. They've made it less obvious. This strange emotional pretend person tugging is a core part of this "product" and its only remarkable feature. Is it enough to justify the gross distortion of the markets? Probably not. But it is ... IDK an innovation in advertising.
I've only found one way to deal with advertising in my life: I block them all.
Other people will claim they are immune to propaganda but I don't believe them.
