Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
-
@cceckman The contract I thought I was signing was this: I published my stuff on a worldwide information network, with no controls whatever, specifically so that anyone anywhere could access it. I did that with full understanding that it would enable people I might not like to read, copy, and share it and put it to uses that I couldn't foresee and might not approve of. And if I didn't want to entertain that possibility I should not have installed a program on my computer whose sole purpose was to deliver of my stuff to any rando who asked for it.
I'm not saying I got a good deal, or that I'm happy with the outcome. But I'm not going to pretend I was tricked or that Google reneged on a bargain. We had no bargain. I served them the stuff anyway, whenever they asked for it.
And I'm not sure I believe Paul Cantrell when he says he thought the contract was different from what I said.
-
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
@inthehands meta noindex it is, definitely. robots disallow can actually hurt the process, since google cannot access the file with the noindex header and therefore won't deindex.
btw, they do indeed respect noindex and robots.txt ATM, since its qute easy to check if pages still get found. Then again, you never know what does not show up in search but is used for training (without giving credit, obv.) anyway. As far as i see, google still remains more standard compliant as e.g. OpenAI. -
Yes, if I wanted everyone to access my online bank account, that's exactly what I would do.
-
And those protections are what, exactly?
They create a legal right to sue for damages (statutory and actual) in federal court. Nothing more.
So sue then.
-
@inthehands I know of at least one professional artist who has deliberately poisoned their images, in an attempt to deter AI scraping (mostly because the scrapers blast her small site and effectively DoS it). If they follow robots.txt, they're not affected... but they were already ignoring robots.txt
I just read an IARPA paper that said poisoning as little as .1% of training data can disrupt a model. If content creators choose to deliberately poison content that they ask not to be scraped, it might be a nice way to deter bad behavior.
The tools I know of work on imagery, but with effort people may come up with stuff that works on data as well. E.g., burying base64-encoded malicious prompts in your text, posting tables as poisoned images rather than text, etc.
Seems like we should start organizing and taking firm action now, before AI companies start buying politicians and making such defenses illegal.
@inthehands And since I saw the question (which was immediately deleted - they probably googled the answer after asking): You use a tool like Nightshade (https://nightshade.cs.uchicago.edu/whatis.html), which modifies the image in a way that's imperceptible to humans, but very visible to AI, effectively making AI "see" the image differently than a human would. When used in AI training, the AI may "see" a toaster when the picture (what a human sees) is actually a photo of a person sitting in a car. When the AI is then asked to generate a picture of someone in a car, it outputs a toaster.
Obviously one image won't do this, but when used at scale it can have an impact.
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands
Given that they've dispensed with the social contract, most websites would pretty much need to implement this through something like nginx, no? I think the signal of updating the robots.txt is important but whatever the broader strategy ends up being, be probably want to assert hard boundaries. -
@inthehands @joe @ShadSterling
Google went out on May 15th and said in their new "spam policy" (seemingly as a preemption) that they will downrank or completely delist sites that try to mess with their AI. Primarily SEO related but I suspect it also extends to "getting it to say stuff" and general poisoning.
I've thought about putting my poison on noindex pages only in order to keep regular search clean and encourage a safe-haven. Might be pointless now.
marginalia-search is pretty nice.
@androidarts @inthehands @ShadSterling the teeth in that threat seem to still be predicated on the website getting actual value from being ranked by Google
-
Yes, if I wanted everyone to access my online bank account, that's exactly what I would do.
-
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands On the one hand, I agree - if nothing else, because caring, doing and building are what I *want* to do.
But on the other hand: "We will rebel against the AI oligarchs by creating *even better* training data for them!"
-
I return
402 Payment Requiredto googlebot user agents@khm @inthehands I wonder if this has legal implications for their bypassing it...
-
@theothersimo @cceckman I didn't suggest “should be”. I will stand by “All information on the WWW is available to malicious actors”.
You seem surprised at this fact, and say the premise is “idiotic”. Okay.
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
Feeling better about having added Googlebot to robots.txt last year
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands I already had blocked them off my Mastodon server (albeit, maybe not fully successful, seems like it still shows links).
-
@inthehands meta noindex it is, definitely. robots disallow can actually hurt the process, since google cannot access the file with the noindex header and therefore won't deindex.
btw, they do indeed respect noindex and robots.txt ATM, since its qute easy to check if pages still get found. Then again, you never know what does not show up in search but is used for training (without giving credit, obv.) anyway. As far as i see, google still remains more standard compliant as e.g. OpenAI.@korrupt @inthehands
Then my question is: Will Google claim that their AI search isn't subject to the old conventions and use that data to train AI and serve those results in their new format? -
@theothersimo @cceckman I didn't suggest “should be”. I will stand by “All information on the WWW is available to malicious actors”.
You seem surprised at this fact, and say the premise is “idiotic”. Okay.
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands I directly block on the webserver using https://perishablepress.com/ultimate-ai-block-list/ with a 403 Forbidden response.
I include anything containing "google" in the list. -
@cceckman The contract I thought I was signing was this: I published my stuff on a worldwide information network, with no controls whatever, specifically so that anyone anywhere could access it. I did that with full understanding that it would enable people I might not like to read, copy, and share it and put it to uses that I couldn't foresee and might not approve of. And if I didn't want to entertain that possibility I should not have installed a program on my computer whose sole purpose was to deliver of my stuff to any rando who asked for it.
I'm not saying I got a good deal, or that I'm happy with the outcome. But I'm not going to pretend I was tricked or that Google reneged on a bargain. We had no bargain. I served them the stuff anyway, whenever they asked for it.
And I'm not sure I believe Paul Cantrell when he says he thought the contract was different from what I said.
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands crawlers choose whether or not they want to oblige robots.txt and meta noindex/nofollow.
The proper way to do this is add agent detection on the server-side, and force a 403. This essentially refuses a request.
This only works if you know all of the agents and they’re not using covert agents. Anyone can use any agent to crawl the web.
But the 403 solution is pretty solid overall.
-
in nginx I have this
if ($http_user_agent ~* (uptime|bot|index|spider|wler|brave)) { return 402 "Just send the money"; }it keeps out the riffraff.
CC: @hyc@mastodon.social @inthehands@hachyderm.io@khm @macronaut @hyc @inthehands I'm considering adding "agent" as one of the options for this regexp...
-
@ShadSterling @mjd @cceckman yeah fair, I only commented because this is one place the distinction matters in that a social contract exists in aggregate as a set of expectations regardless of what an individual might expect or feel like they agreed to
@wronglang @mjd @cceckman right, which is distinct enough that it would be better to have a more distinct name for it
