Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands How about blocking port 80 & 443 access from Google's netblocks? 142.250.0.0/15 for starters.
-
@hyc @khm @inthehands is there a “how to” on this that one can use to update their web server/site?
@macronaut @khm @inthehands Currently I've created in nginx/conf/server_extra/block-useragent.conf:
if ($http_user_agent ~* meta-externalagent) {
return 403;
}And I've added an
include server_extra/*.conf;in my site's server{} config.
-
@macronaut @khm @inthehands Currently I've created in nginx/conf/server_extra/block-useragent.conf:
if ($http_user_agent ~* meta-externalagent) {
return 403;
}And I've added an
include server_extra/*.conf;in my site's server{} config.
@macronaut @khm @inthehands the update would change that 403 to a 402. And add "error_page 402 /402.html;" to the server{} config, and create the /402.html file in the docroot containing whatever desired message.
-
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands Dissallow in robots.txt and install iocaine or anubis or another AI-poisoning software, should they ignore the robots.txt.
-
@cceckman The contract I thought I was signing was this: I published my stuff on a worldwide information network, with no controls whatever, specifically so that anyone anywhere could access it. I did that with full understanding that it would enable people I might not like to read, copy, and share it and put it to uses that I couldn't foresee and might not approve of. And if I didn't want to entertain that possibility I should not have installed a program on my computer whose sole purpose was to deliver of my stuff to any rando who asked for it.
I'm not saying I got a good deal, or that I'm happy with the outcome. But I'm not going to pretend I was tricked or that Google reneged on a bargain. We had no bargain. I served them the stuff anyway, whenever they asked for it.
And I'm not sure I believe Paul Cantrell when he says he thought the contract was different from what I said.
-
@hyc @khm @inthehands is there a “how to” on this that one can use to update their web server/site?
in nginx I have thisif ($http_user_agent ~* (uptime|bot|index|spider|wler|brave)) { return 402 "Just send the money"; }it keeps out the riffraff.
CC: @hyc@mastodon.social @inthehands@hachyderm.io
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands They're just going to take it anyway.
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands Reciprocate how?
-
@macronaut @khm @inthehands Currently I've created in nginx/conf/server_extra/block-useragent.conf:
if ($http_user_agent ~* meta-externalagent) {
return 403;
}And I've added an
include server_extra/*.conf;in my site's server{} config.
@hyc@mastodon.social @macronaut@mas.to @khm@hj.9fs.net @inthehands@hachyderm.io
I redirect those who make some requests either to FSB or to CIA depending on particular paths.
Sadly, I don't think scripts making those requests follow redirects, but those who request .php files which I don't have on my server get redirected to FSB website
-
@joe @ShadSterling
I share Joe’s concern that poison-in-box systems will become detectable, but they seem like a good place to start.I’m even more a fan of bespoke one-off poison generators for those of us who have the means to write them. Both/and.
@inthehands @joe @ShadSterling
Google went out on May 15th and said in their new "spam policy" (seemingly as a preemption) that they will downrank or completely delist sites that try to mess with their AI. Primarily SEO related but I suspect it also extends to "getting it to say stuff" and general poisoning.
I've thought about putting my poison on noindex pages only in order to keep regular search clean and encourage a safe-haven. Might be pointless now.
marginalia-search is pretty nice.
-
@cceckman The contract I thought I was signing was this: I published my stuff on a worldwide information network, with no controls whatever, specifically so that anyone anywhere could access it. I did that with full understanding that it would enable people I might not like to read, copy, and share it and put it to uses that I couldn't foresee and might not approve of. And if I didn't want to entertain that possibility I should not have installed a program on my computer whose sole purpose was to deliver of my stuff to any rando who asked for it.
I'm not saying I got a good deal, or that I'm happy with the outcome. But I'm not going to pretend I was tricked or that Google reneged on a bargain. We had no bargain. I served them the stuff anyway, whenever they asked for it.
And I'm not sure I believe Paul Cantrell when he says he thought the contract was different from what I said.
@mjd You thought posting something on the internet makes it public domain?
-
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands
I doubt the cynicism = surrender part. Cynicism is refusing to surrender in the face of an overly mighty enemy. -
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands I wonder if we need to go back to the things the companies used to use to hold us captive to keep them out. Walled gardens for instance. Something like a web list or web circle behind a login.
-
I return
402 Payment Requiredto googlebot user agents@khm @inthehands that's awesome. i like iocaine as another way to mess with the bots. Give them 200's but serve them garbage
-
@mjd You thought posting something on the internet makes it public domain?
@williamoconnell I didn't think that. But I did understand that as a practical matter it meant relinquishing most of my control over it.
-
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands but it's haaaaaaard
-
@cceckman The contract I thought I was signing was this: I published my stuff on a worldwide information network, with no controls whatever, specifically so that anyone anywhere could access it. I did that with full understanding that it would enable people I might not like to read, copy, and share it and put it to uses that I couldn't foresee and might not approve of. And if I didn't want to entertain that possibility I should not have installed a program on my computer whose sole purpose was to deliver of my stuff to any rando who asked for it.
I'm not saying I got a good deal, or that I'm happy with the outcome. But I'm not going to pretend I was tricked or that Google reneged on a bargain. We had no bargain. I served them the stuff anyway, whenever they asked for it.
And I'm not sure I believe Paul Cantrell when he says he thought the contract was different from what I said.
-
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
@inthehands meta noindex it is, definitely. robots disallow can actually hurt the process, since google cannot access the file with the noindex header and therefore won't deindex.
btw, they do indeed respect noindex and robots.txt ATM, since its qute easy to check if pages still get found. Then again, you never know what does not show up in search but is used for training (without giving credit, obv.) anyway. As far as i see, google still remains more standard compliant as e.g. OpenAI. -
Yes, if I wanted everyone to access my online bank account, that's exactly what I would do.
