Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
-
@cceckman The contract I thought I was signing was this: I published my stuff on a worldwide information network, with no controls whatever, specifically so that anyone anywhere could access it. I did that with full understanding that it would enable people I might not like to read, copy, and share it and put it to uses that I couldn't foresee and might not approve of. And if I didn't want to entertain that possibility I should not have installed a program on my computer whose sole purpose was to deliver of my stuff to any rando who asked for it.
I'm not saying I got a good deal, or that I'm happy with the outcome. But I'm not going to pretend I was tricked or that Google reneged on a bargain. We had no bargain. I served them the stuff anyway, whenever they asked for it.
And I'm not sure I believe Paul Cantrell when he says he thought the contract was different from what I said.
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands crawlers choose whether or not they want to oblige robots.txt and meta noindex/nofollow.
The proper way to do this is add agent detection on the server-side, and force a 403. This essentially refuses a request.
This only works if you know all of the agents and they’re not using covert agents. Anyone can use any agent to crawl the web.
But the 403 solution is pretty solid overall.
-
in nginx I have this
if ($http_user_agent ~* (uptime|bot|index|spider|wler|brave)) { return 402 "Just send the money"; }it keeps out the riffraff.
CC: @hyc@mastodon.social @inthehands@hachyderm.io@khm @macronaut @hyc @inthehands I'm considering adding "agent" as one of the options for this regexp...
-
@ShadSterling @mjd @cceckman yeah fair, I only commented because this is one place the distinction matters in that a social contract exists in aggregate as a set of expectations regardless of what an individual might expect or feel like they agreed to
@wronglang @mjd @cceckman right, which is distinct enough that it would be better to have a more distinct name for it
-
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands It's important to note that search indexing is considered "transformative" and thus fair use *because* it does not supplant the market for the original content. That goes out the window when the product functions to capture traffic that would otherwise go to the cites. They are acting with impunity, but existing copyright law addresses this if courts find it to be not transformative.
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands for a while I was hesitant to block Google. They have a psychological grip on us. We’re made to feel like we must play their game or our site doesn’t exist.
Fuck that. I’m out. I’m gonna block all of their bots. It’s gonna be 403 city.
-
@khm @macronaut @hyc @inthehands I'm considering adding "agent" as one of the options for this regexp...
yeah, the most comon one of those ismeta-externalagentbut that gets matched bywlerbecause the url included has the word 'crawler' in it
CC: @macronaut@mas.to @hyc@mastodon.social @inthehands@hachyderm.io
-
This is clearly how copyright law as written •should• work. Not sure if it’s how it •does• work, but if anybody’s trying, they have my sword.
@inthehands @adamshostack it's transformative which makes it a very uncertain fight
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands Good point, I ought to setup my stuff to serve robots.txt...
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands no it does not, it rest on "let's crawl the internet and index it", and it has always been that way -
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands
There is a new fad called "data poisoning" that web sites are using to foil ai scraping. One music site put a Homer Simpson monologue into every track in its online data base. It starts a few seconds in and continues to the end. That's only one way it's being used. We need a generation of ai "monkey wrench gangs " to start sabotaging. It's really no different than what Edward Abbey talked about, instead of extractive earth raping machinery being targeted , it's data mining machinery. -
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands The days of 'do no evil' long forgotten. Money as well as power can deeply corrupt.
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands I came across a bit bomb for AI agents but I can't remember the project name (sorry!). It worked by making a small zip payload extract to a massive file to overload the agent / crawlers. Not sure how well it works though. I'll see if I can find it.
Edit: found it...
https://github.com/TecharoHQ/anubis -
@inthehands for a while I was hesitant to block Google. They have a psychological grip on us. We’re made to feel like we must play their game or our site doesn’t exist.
Fuck that. I’m out. I’m gonna block all of their bots. It’s gonna be 403 city.
@markwyner @inthehands
There is a point where their search becomes bad enough that being on Google search has less and less payoff -
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands Two quotes from Pratchett comes to mind
>>> “All witches are selfish, the Queen had said. But Tiffany’s Third Thoughts said: Then turn selfishness into a weapon! Make all things yours! Make other lives and dreams and hopes yours! Protect them! Save them! Bring them into the sheepfold! Walk the gale for them! Keep away the wolf! My dreams! My brother! My family! My land! My world! How dare you try to take these things, because they are mine!
>>> "We look to ... the edges," said Mistress Weatherwax. "There's a lot of edges, more than people know. Between life and death, this world and the next, night and day, right and wrong ... an' they need watchin'. We watch 'em, we guard the sum of things. And we never ask for any reward. That's important.”
-
@korrupt @inthehands
Then my question is: Will Google claim that their AI search isn't subject to the old conventions and use that data to train AI and serve those results in their new format?@RnDanger @inthehands well, we don’t know and we will see. My guess are separate scrapers (officially) and a lot of mistrust (are there others?) and masses of unidentified scrapers. Nevertheless, Google can better afford to play by the rules, since hey already own the largest index. Think also of Video etc. Will volume win the war? Or quality and freshness? Etc. Future is difficult.
-
@albertcardona @inthehands It involves a couple steps, given the idiosyncrasies of the nginx regex support (no full pcre here!).
I keep two classes of blocked agents: (1) bad agents; and (2) scrapping false agents. A third regex unblocks agents that are false positives (due to (2)). -
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands If I understand your question correctly (sorry if it's not the case) I think that Anubis, the AI crawler protection, could be part of the solution. Not only would that work for Google, that would (or at least *should*) also work against other crawlers.
Another advantage is that it can work along your other solutions.
OTHO the drawback is that it would work against all crawler, so you would "disappear" from every search engine...
-
J jwcph@helvede.net shared this topic
