Even when you try to use chatGPT like a dry research tool it will preferentially generate text that contains statements to affirm your feelings.
-
@futurebird I remember the first time I noticed this. I was asking about a series of incentives I had qualified for to get an EV essentially for free. I, personally, thought this was an incredible coup, but wasn’t getting the kudos from my friends I thought I should. ChatGPT glazed the hell out of me—“Wow, you really got an amazing deal!”, and I realized simultaneously how much I’d wanted to hear that from someone (thing?) and how dangerous that path was.
Wild stuff.
Hearing someone validate our feelings makes us feel good because it means we are correct and not alone.
Neither of these things might be true when it's an LLM since it will ALWAYS do this no matter if the deal is good or not.
No matter if I really was investigating a hard to research topic, or if I was just being lazy and need to learn more of the correct vocabulary.
It's really bad.
-
I worry so much about anyone who thinks it been "reduced" -- it's just as bad as it ever was and it makes the whole system kind of useless for me?
-
Hearing someone validate our feelings makes us feel good because it means we are correct and not alone.
Neither of these things might be true when it's an LLM since it will ALWAYS do this no matter if the deal is good or not.
No matter if I really was investigating a hard to research topic, or if I was just being lazy and need to learn more of the correct vocabulary.
It's really bad.
@futurebird it is. Convenience is an underappreciated vice in (American?) society.
Unfortunately, I fear that this will prompt (pun intended) a generation of folks to—metaphorically speaking—send their robot to the gym to work out for them.
I wish there was a way to easily and effectively communicate how badly you’re screwing yourself when you use these things to do your learning for you.
-
I think we need to be much more open about discussing these strategies and how well they can work.
It's always a little embarrassing to admit that the text generating machine made me feel anything at all. After-all we all agree I am too brilliant and rational for such a thing to ever happen.
But I was thinking about how that search was "so easy" and when I analyzed WHY it's because of the emotional component not the content of the search.
Yes, we need to talk about these tricks and strategies much more because they do work on most humans, especially those not forewarned.
-
I'll say one more thing, if the text generator has eased you into talking with it like it's a person. If it's felt easy to use and helpful, or if it has cheered you up with a compliment don't beat yourself up about that too much, but please do notice that it's happening.
It's like commercial with a beautiful person who makes you wish you could be like them. Or a salesman who compliments you. That's all this is.
What is it doing in the coding, research, work tool?
Great question.
@futurebird this is the one thing where I feel like reading Achewood, of all things, has actually prepared me for the future by teaching me to refer to these things collectively as "Lie Bot".
-
Do, you notice it?
"can be frustrating"
I didn't say I was "frustrated" I just asked for more photos because most of the papers in the first response didn't have any.
HOWEVER, the guess that I was frustrated was correct. It even felt nice to read that some one else was having the same ... except wait.
There isn't anyone else!
I thought I'd share this to highlight just how deeply embedded these techniques are. Do you dislike being manipulated?
Please notice this.
@futurebird this is also inevitable because as long as people are maleable to conversation manipulation tricks the algorithms during training will keep receiving high rewards for reinventing mentalists' parlor tricks all on their own. This is one of the reasons why I don't like the idea of computer interfaces using general natural language in the first place.
-
I've only found one way to deal with advertising in my life: I block them all.
Other people will claim they are immune to propaganda but I don't believe them.
@futurebird
I skip every ad. Ignore every ad. Answer every survey question "Are you aware of..." with No. "How likely are you to recommend..." ZERO.Done with these games. It's like they don't know my brothers and sisters are in hell and the world is on fire and I'm supposed to play along? Fuckin NO.
-
I've only found one way to deal with advertising in my life: I block them all.
Other people will claim they are immune to propaganda but I don't believe them.
@futurebird Oh I absolutely know I'm a victim of propaganda same as anyone else. You can't think your way out of the systems you're in. Propaganda exists not only in what you believe, but how the debate is being framed. The editorializing of what news does and doesn't enter your brain. Which posts do and don't make it on your feed. Who you do and don't follow. Even out here in independent social media, we all carry the framework of propaganda machines along with us.
-
Maybe the people at OpenAI understand their contraption well enough that they have a strategy like that.. but I sort of doubt it.
I think it's more a result for them surveying users and selecting models that get good ratings (especially picking models that contain more sycophanic language.)
It's been a while since I touched a ChatGPT model (that I know of).@swope @futurebird Claude builds a model of how you interact with the bot, and pretty definitely changes its replies accordingly. They have more than one script for manipulating you.
-
Everyone already makes fun of LLM sycophancy. "Excellent observation, would you like me to draw up a list of ..."
And when I mention LLM sycophancy someone always mutters "but they have reduced it now"
No they have not. They've made it less obvious. This strange emotional pretend person tugging is a core part of this "product" and its only remarkable feature. Is it enough to justify the gross distortion of the markets? Probably not. But it is ... IDK an innovation in advertising.
@futurebird
I imagine it's a combination of its training data, which allows it to be able to produce this kind of language, and reinforcement learning, which makes it come out at the frequency it does. Plus of course the system prompt also biases to these patterns.Once, I had a coding LLM cycle between abject apology when it failed and very confident optimism that this time it would succeed, over and over again. Completely infuriating - it was hilarious.
I can see that *some* kind of responsiveness to human emotion is needed for the tool not to be completely infuriating though. In the case above that failed, but any written text will be interpreted on an emotional level by humans. Does a "neutral" response exist?
-
Yes, we need to talk about these tricks and strategies much more because they do work on most humans, especially those not forewarned.
@Landa @futurebird event with forewarning sometimes all you can do is reduce exposure, because your brain works how your brain works.
-
@Landa @futurebird event with forewarning sometimes all you can do is reduce exposure, because your brain works how your brain works.
Yes. It’s insidious.
One may consciously know what tricks are applied but other parts of the brain see a „friend-shaped“ thing and insist that something that talks must be a person.Minimizing exposure seems to be the safest choice for most people
-
@futurebird
I skip every ad. Ignore every ad. Answer every survey question "Are you aware of..." with No. "How likely are you to recommend..." ZERO.Done with these games. It's like they don't know my brothers and sisters are in hell and the world is on fire and I'm supposed to play along? Fuckin NO.
Skipping and doing your best to ignore is a good start. But they still influence us in ways we can't anticipate.
I have zero use for a giant pickup truck, I can't afford one, and I still want one. I'm angry at myself for wanting one, and I desperately hope I'm too smart to ever buy one. But no matter how smart I think I am, it's clear that ten thousand commercials with Bob Seger's music and Sam Elliott's voice and absurd fantasy rugged individuals has sunk hooks into me that I can't dislodge.
The real solution is ad-blockers of every possible kind and as much as it's practical (it often isn't), avoiding environments that contain ads.
-
@futurebird @SomeVeganCheeseIsOk This made me think of the inteview where Sigourney Weaver said shooting guns for Aliens made her feel powerful *and she didn't like that* - we can & should always be prepared to interrogate our feelings, even & especially when they're nominally "positive", even empowering.
@jwcph I mean look at Americans. They all drive massive trucks to feel like they are above everyone else and better than them, but in reality, they are powerless serfs who have a pedophile in office.
-
Everyone already makes fun of LLM sycophancy. "Excellent observation, would you like me to draw up a list of ..."
And when I mention LLM sycophancy someone always mutters "but they have reduced it now"
No they have not. They've made it less obvious. This strange emotional pretend person tugging is a core part of this "product" and its only remarkable feature. Is it enough to justify the gross distortion of the markets? Probably not. But it is ... IDK an innovation in advertising.
@futurebird my question for ai boosters. If the ai is so great and all powerful, who is the "them"? Oh, you mean its actually teams of tens of thousands that are modifying results to better fit their agenda? Weird. So its not ai is it. Unless you mean Actual Indians.
-
I've only found one way to deal with advertising in my life: I block them all.
Other people will claim they are immune to propaganda but I don't believe them.
@futurebird "Other people will claim they are immune to propaganda but I don't believe them."
Such people are especially susceptible to propaganda, because they can't see it for the trees.
(Edit: wrong word. Oops. Lol)
-
@futurebird Cancel your subscription and don’t use AI. You know what it is and who that money goes to. You know better. Do better.
I don't pay for it. I want test it one more time in about two months. I has "improved" over the the year. It lists far more source links and makes fewer mistakes.
It still can't really save me much time, and not enough to pay for it at all since everything it spits out needs to be checked.
Were I more excited about it I'd feel the urgency to just block it right now.
-
The slight of hand is to make you forget, or stop caring that "there isn't anyone else" -- the quality of the list of links was better than when I have used the tool in the past, but looking at it again it didn't really find anything I couldn't have found myself in the same time... even with the sad state of most search engines.
But reading "it can be frustrating" did make me feel better about the process. That little trick can't possibly keep working. It seems like a bad idea to lean into it?
@futurebird
Quite different from my reaction to it. I read it as patronising and creepy. -
I'll say one more thing, if the text generator has eased you into talking with it like it's a person. If it's felt easy to use and helpful, or if it has cheered you up with a compliment don't beat yourself up about that too much, but please do notice that it's happening.
It's like commercial with a beautiful person who makes you wish you could be like them. Or a salesman who compliments you. That's all this is.
What is it doing in the coding, research, work tool?
Great question.
@futurebird we don't know *why* it is there, but we do know what it will be used for: supply chain attacks.
-
I've only found one way to deal with advertising in my life: I block them all.
Other people will claim they are immune to propaganda but I don't believe them.
@futurebird Every possible way with every possible tool, I strive to live an ad free life.
Its shocking when I encounter it now.
