Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
Same question as the previous post, except for Wkipedia. What would you like to see them do to send a shot across the bow?
Or…well, it’s Wikipedia. Maybe more like a shot to the hull.
3/2
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands (3) sue on the basis that’s it’s not fair use, and these derivative works clearly have a dramatic impact on the value of the original site
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands is "serve LLM poison to googlebot user-agents" on the table
-
@inthehands (3) sue on the basis that’s it’s not fair use, and these derivative works clearly have a dramatic impact on the value of the original site
This is clearly how copyright law as written •should• work. Not sure if it’s how it •does• work, but if anybody’s trying, they have my sword.
-
@inthehands is "serve LLM poison to googlebot user-agents" on the table
@joe
It is and some of us miiiiight already be doing it. -
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands both, and, when the agent matches the heuristics to be recognized as Google (et. al.), send a different response that contains only an explanation of the ban (and maybe some poison for their next model)
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@cceckman The contract I thought I was signing was this: I published my stuff on a worldwide information network, with no controls whatever, specifically so that anyone anywhere could access it. I did that with full understanding that it would enable people I might not like to read, copy, and share it and put it to uses that I couldn't foresee and might not approve of. And if I didn't want to entertain that possibility I should not have installed a program on my computer whose sole purpose was to deliver of my stuff to any rando who asked for it.
I'm not saying I got a good deal, or that I'm happy with the outcome. But I'm not going to pretend I was tricked or that Google reneged on a bargain. We had no bargain. I served them the stuff anyway, whenever they asked for it.
And I'm not sure I believe Paul Cantrell when he says he thought the contract was different from what I said.
-
@cceckman The contract I thought I was signing was this: I published my stuff on a worldwide information network, with no controls whatever, specifically so that anyone anywhere could access it. I did that with full understanding that it would enable people I might not like to read, copy, and share it and put it to uses that I couldn't foresee and might not approve of. And if I didn't want to entertain that possibility I should not have installed a program on my computer whose sole purpose was to deliver of my stuff to any rando who asked for it.
I'm not saying I got a good deal, or that I'm happy with the outcome. But I'm not going to pretend I was tricked or that Google reneged on a bargain. We had no bargain. I served them the stuff anyway, whenever they asked for it.
And I'm not sure I believe Paul Cantrell when he says he thought the contract was different from what I said.
-
Same question as the previous post, except for Wkipedia. What would you like to see them do to send a shot across the bow?
Or…well, it’s Wikipedia. Maybe more like a shot to the hull.
3/2
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands if they decide that people doing this hurts their business model they will simply stop respecting things like robots.txt. their gamble is that people rely on Google more than they do on other websites and if they have to kill the rest of the web to monopolize access to information they will.
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands while this can (and probably should) be done in tandem with other strategies, one of the most unambiguous ways you can express your disdain is in robots.txt. Google has historically respected it mechanically (in the present and future, I'm not sure this will hold), and it supports line comments with # so you can explain in plain English what you think about them.
https://developers.google.com/search/docs/crawling-indexing/robots/intro
The docs also mention the 'noindex' meta tag and how you probably want to use one or the other but not both. That's worth a little research, probably.
-
@inthehands if they decide that people doing this hurts their business model they will simply stop respecting things like robots.txt. their gamble is that people rely on Google more than they do on other websites and if they have to kill the rest of the web to monopolize access to information they will.
Of course, but it is important to force that fight rather than capitulating in advance.
-
@joe
It is and some of us miiiiight already be doing it.@inthehands given how eager their summarizer is to incorporate "facts" from even unintentionally adversarial recent posts like satirical blogs, it seems like it wouldn't take much of a coordinated effort to reduce their result quality this way
-
@wronglang @mjd @cceckman this sort of discrepancy is why I’ve never liked the term “social contract” - it’s nothing like a “contract”
-
@inthehands given how eager their summarizer is to incorporate "facts" from even unintentionally adversarial recent posts like satirical blogs, it seems like it wouldn't take much of a coordinated effort to reduce their result quality this way
@joe @inthehands is there a coordinated effort that has a website? And/or server plugins that automate serving coordinated poison?
-
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
@inthehands this is a fence-post defense against this, google Will Not Care
just start poisoning the data once you detect that google is the one fetching it, just absolutely fucking destroy their LLM output
-
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
@inthehands also probably worth it to submit a pagemaster/webmaster request to them to directly tell them to deindex your site. Also DMCA takedowns to Google are usually effective. If you're in the jurisdiction of Australia you're potentially able to go after them iirc. (The Australian government went after them for embedding news articles in their output or something)
-
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
@inthehands
What guarantee does one have that Google will abide by these restrictions? -
@cceckman The contract I thought I was signing was this: I published my stuff on a worldwide information network, with no controls whatever, specifically so that anyone anywhere could access it. I did that with full understanding that it would enable people I might not like to read, copy, and share it and put it to uses that I couldn't foresee and might not approve of. And if I didn't want to entertain that possibility I should not have installed a program on my computer whose sole purpose was to deliver of my stuff to any rando who asked for it.
I'm not saying I got a good deal, or that I'm happy with the outcome. But I'm not going to pretend I was tricked or that Google reneged on a bargain. We had no bargain. I served them the stuff anyway, whenever they asked for it.
And I'm not sure I believe Paul Cantrell when he says he thought the contract was different from what I said.
@mjd (thanks- this made me write down some of the thoughts I've been kicking around for a bit!)
"publish freely and for free" is also my site's situation, but it is a privileged one.
My mom's website is a funnel for her books, courses, services, etc. It is useful for her to publish helpful things for people to learn from; and to have those indexed in search; so that when people look for an answer to a question, they find out about her, and may wind up giving her money in the future.
Someone *can* scrape what she writes and re-post it, sure. And that's (in my opinion) a bad thing to do, because now the person who has done the work doesn't get any benefits from it: reputation, income, whatever. People *can* do that- SEO farms, image-sharing sites, whatnot- but as long as there is enough traffic going to the author/creator, there's still a viable business in "write for the web". (There is at least some legal consensus around this idea in the form of copyright law- that an author should be compensated for their work.)
What happens to that business model if "search" leads, not to the author's site, but to a hologram of it? Without providing the author any benefit, in citation (reputation), ad revenue, or sales funnel?
Some authors (like you, maybe? and like me?) will still write and publish. Wikipedia editors will still edit! The web doesn't fully disappear.
But...the incentives to provide that information *professionally* are degraded; the income has to come from somewhere else. And there's a lot of bad ideas that have better funding than some good ones.
If Anthropic scrapes the Crochet Answer Book (they did) and answers questions from it (it does), then who is going to write the next edition? Are Anthropic, Google, OpenAI going to pay for the one copy each they need to buy? Or is it necessarily the effort of someone who is not fairly paid for it?
If we get fully automated luxury space communism, fine, share everything for free; but, in the world we're in-- I'm worried that there's no longer a *financial* incentive to share reliable information openly. So some of those sources will dry up.
