Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
-
Same question as the previous post, except for Wkipedia. What would you like to see them do to send a shot across the bow?
Or…well, it’s Wikipedia. Maybe more like a shot to the hull.
3/2
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands if they decide that people doing this hurts their business model they will simply stop respecting things like robots.txt. their gamble is that people rely on Google more than they do on other websites and if they have to kill the rest of the web to monopolize access to information they will.
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands while this can (and probably should) be done in tandem with other strategies, one of the most unambiguous ways you can express your disdain is in robots.txt. Google has historically respected it mechanically (in the present and future, I'm not sure this will hold), and it supports line comments with # so you can explain in plain English what you think about them.
https://developers.google.com/search/docs/crawling-indexing/robots/intro
The docs also mention the 'noindex' meta tag and how you probably want to use one or the other but not both. That's worth a little research, probably.
-
@inthehands if they decide that people doing this hurts their business model they will simply stop respecting things like robots.txt. their gamble is that people rely on Google more than they do on other websites and if they have to kill the rest of the web to monopolize access to information they will.
Of course, but it is important to force that fight rather than capitulating in advance.
-
@joe
It is and some of us miiiiight already be doing it.@inthehands given how eager their summarizer is to incorporate "facts" from even unintentionally adversarial recent posts like satirical blogs, it seems like it wouldn't take much of a coordinated effort to reduce their result quality this way
-
@wronglang @mjd @cceckman this sort of discrepancy is why I’ve never liked the term “social contract” - it’s nothing like a “contract”
-
@inthehands given how eager their summarizer is to incorporate "facts" from even unintentionally adversarial recent posts like satirical blogs, it seems like it wouldn't take much of a coordinated effort to reduce their result quality this way
@joe @inthehands is there a coordinated effort that has a website? And/or server plugins that automate serving coordinated poison?
-
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
@inthehands this is a fence-post defense against this, google Will Not Care
just start poisoning the data once you detect that google is the one fetching it, just absolutely fucking destroy their LLM output
-
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
@inthehands also probably worth it to submit a pagemaster/webmaster request to them to directly tell them to deindex your site. Also DMCA takedowns to Google are usually effective. If you're in the jurisdiction of Australia you're potentially able to go after them iirc. (The Australian government went after them for embedding news articles in their output or something)
-
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
@inthehands
What guarantee does one have that Google will abide by these restrictions? -
@cceckman The contract I thought I was signing was this: I published my stuff on a worldwide information network, with no controls whatever, specifically so that anyone anywhere could access it. I did that with full understanding that it would enable people I might not like to read, copy, and share it and put it to uses that I couldn't foresee and might not approve of. And if I didn't want to entertain that possibility I should not have installed a program on my computer whose sole purpose was to deliver of my stuff to any rando who asked for it.
I'm not saying I got a good deal, or that I'm happy with the outcome. But I'm not going to pretend I was tricked or that Google reneged on a bargain. We had no bargain. I served them the stuff anyway, whenever they asked for it.
And I'm not sure I believe Paul Cantrell when he says he thought the contract was different from what I said.
@mjd (thanks- this made me write down some of the thoughts I've been kicking around for a bit!)
"publish freely and for free" is also my site's situation, but it is a privileged one.
My mom's website is a funnel for her books, courses, services, etc. It is useful for her to publish helpful things for people to learn from; and to have those indexed in search; so that when people look for an answer to a question, they find out about her, and may wind up giving her money in the future.
Someone *can* scrape what she writes and re-post it, sure. And that's (in my opinion) a bad thing to do, because now the person who has done the work doesn't get any benefits from it: reputation, income, whatever. People *can* do that- SEO farms, image-sharing sites, whatnot- but as long as there is enough traffic going to the author/creator, there's still a viable business in "write for the web". (There is at least some legal consensus around this idea in the form of copyright law- that an author should be compensated for their work.)
What happens to that business model if "search" leads, not to the author's site, but to a hologram of it? Without providing the author any benefit, in citation (reputation), ad revenue, or sales funnel?
Some authors (like you, maybe? and like me?) will still write and publish. Wikipedia editors will still edit! The web doesn't fully disappear.
But...the incentives to provide that information *professionally* are degraded; the income has to come from somewhere else. And there's a lot of bad ideas that have better funding than some good ones.
If Anthropic scrapes the Crochet Answer Book (they did) and answers questions from it (it does), then who is going to write the next edition? Are Anthropic, Google, OpenAI going to pay for the one copy each they need to buy? Or is it necessarily the effort of someone who is not fairly paid for it?
If we get fully automated luxury space communism, fine, share everything for free; but, in the world we're in-- I'm worried that there's no longer a *financial* incentive to share reliable information openly. So some of those sources will dry up.
-
@joe
It is and some of us miiiiight already be doing it.@inthehands @joe gotta update my robots.txt real quick
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands IMO anything besides active denial measures (proof of work e.g. Anubis) is pointless. These companies don’t care about consent, and if they do that care is completely predicated on a few important nerds being in positions of power. That won’t last forever, especially in an environment where a larger and larger number of software engineers are people cashing a check rather than people who have a values based commitment to an open internet.
-
@wronglang @mjd @cceckman this sort of discrepancy is why I’ve never liked the term “social contract” - it’s nothing like a “contract”
@ShadSterling @mjd @cceckman yeah fair, I only commented because this is one place the distinction matters in that a social contract exists in aggregate as a set of expectations regardless of what an individual might expect or feel like they agreed to
-
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
-
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
-
@joe @inthehands is there a coordinated effort that has a website? And/or server plugins that automate serving coordinated poison?
@ShadSterling @inthehands i don't know if there's a coordinated movement. there are prefab tools like https://lib.rs/crates/iocaine that are relatively easy to deploy, though i imagine they also lose some of their effectiveness as they become more popular and LLM providers start to counter them
-
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands @iwein HEAR HEAR!!
-
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
@inthehands One of the things I've done recently is to bring enforcing robots.txt within my webserver engine. The /robots.txt itself still exists; the vast majority of it is a list of bots that are `Disallow: /` .
I still get a few of these bots attempting to hit the site, so it's definitely doing something.
-
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
@inthehands you can block their bots at the network level
