Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands IMO anything besides active denial measures (proof of work e.g. Anubis) is pointless. These companies don’t care about consent, and if they do that care is completely predicated on a few important nerds being in positions of power. That won’t last forever, especially in an environment where a larger and larger number of software engineers are people cashing a check rather than people who have a values based commitment to an open internet.
-
@wronglang @mjd @cceckman this sort of discrepancy is why I’ve never liked the term “social contract” - it’s nothing like a “contract”
@ShadSterling @mjd @cceckman yeah fair, I only commented because this is one place the distinction matters in that a social contract exists in aggregate as a set of expectations regardless of what an individual might expect or feel like they agreed to
-
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
-
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
-
@joe @inthehands is there a coordinated effort that has a website? And/or server plugins that automate serving coordinated poison?
@ShadSterling @inthehands i don't know if there's a coordinated movement. there are prefab tools like https://lib.rs/crates/iocaine that are relatively easy to deploy, though i imagine they also lose some of their effectiveness as they become more popular and LLM providers start to counter them
-
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands @iwein HEAR HEAR!!
-
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
@inthehands One of the things I've done recently is to bring enforcing robots.txt within my webserver engine. The /robots.txt itself still exists; the vast majority of it is a list of bots that are `Disallow: /` .
I still get a few of these bots attempting to hit the site, so it's definitely doing something.
-
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
@inthehands you can block their bots at the network level
-
@ShadSterling @inthehands i don't know if there's a coordinated movement. there are prefab tools like https://lib.rs/crates/iocaine that are relatively easy to deploy, though i imagine they also lose some of their effectiveness as they become more popular and LLM providers start to counter them
@joe @ShadSterling
I share Joe’s concern that poison-in-box systems will become detectable, but they seem like a good place to start.I’m even more a fan of bespoke one-off poison generators for those of us who have the means to write them. Both/and.
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands I guess “Search Engine Optimisation” has gained a new, contrary meaning.
-
@joe @ShadSterling
I share Joe’s concern that poison-in-box systems will become detectable, but they seem like a good place to start.I’m even more a fan of bespoke one-off poison generators for those of us who have the means to write them. Both/and.
@inthehands @ShadSterling if you can live with exposing yourself to the demon core a bit, using a locally hosted LLM to generate the poison also seems like a good way to get a lot of hard-to-detect variability, and hopefully also slightly accelerate model collapse in the process
-
@inthehands I guess “Search Engine Optimisation” has gained a new, contrary meaning.
“search engine nopetimization”
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands Hey, I'm curious if you have an opinion on what that contract *should* say? I don't think "pay with traffic" has worked for a long time. What would?
-
@inthehands As I said just a while ago: Every big tech press event these last few years have felt like "Announcing our exciting plans for oligarchs to strip-mine the entire world and immiserate all of humanity! Get on board, and also death to the unbelievers!"
My recommendation is always "Follow the Money" .
Google is now an adjunct of the fossil fuel industry & its fossil fuel funded public corruption.Go after the wealth of the billionaires & oil oligarchs funding Google's #Enshittification
#PrinceBonesaw
Alwaleed bin Talal
Chris Hohn
Elon Musk
Sergey Brin
Peter Thiel
Larry Ellison
Charles Kochhttps://www.washingtonpost.com/technology/2025/05/13/trump-tech-execs-riyadh/
https://www.sfgate.com/tech/article/billionaire-hohn-more-google-layoffs-17736530.php
Fossil fuel phase out.
-
This is clearly how copyright law as written •should• work. Not sure if it’s how it •does• work, but if anybody’s trying, they have my sword.
It almost certainly is but that means going to court. I've not yet seen anyone trying.
Someone who can demonstrate avtual financial damages absolutely should. Especially with the explicit statements of intent from Google now.
-
@inthehands @ShadSterling if you can live with exposing yourself to the demon core a bit, using a locally hosted LLM to generate the poison also seems like a good way to get a lot of hard-to-detect variability, and hopefully also slightly accelerate model collapse in the process
@joe @inthehands @ShadSterling i like to prop the demon core open with a screwdriver and point the aperture at the internet
-
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
@inthehands All this, and a little more! When Google *does* start ignoring robots.txt and other mechanisms, that's another victory for us, not them, even if it means we have to react to it.
Not all of Google's infrastructure is servers in a giant building, or software systems running on top of it, or even offices full of stressed out tech workers. Part of their infrastructure, the cladding on the castle walls, is their false pretense of being good citizens on the internet. When we call their bluff and they eventually drop the pretense, that's us getting them to tear down the outer layers of the castle themselves. We know what they are, and we can make them admit it, and that's power.
-
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
@inthehands I know of at least one professional artist who has deliberately poisoned their images, in an attempt to deter AI scraping (mostly because the scrapers blast her small site and effectively DoS it). If they follow robots.txt, they're not affected... but they were already ignoring robots.txt
I just read an IARPA paper that said poisoning as little as .1% of training data can disrupt a model. If content creators choose to deliberately poison content that they ask not to be scraped, it might be a nice way to deter bad behavior.
The tools I know of work on imagery, but with effort people may come up with stuff that works on data as well. E.g., burying base64-encoded malicious prompts in your text, posting tables as poisoned images rather than text, etc.
Seems like we should start organizing and taking firm action now, before AI companies start buying politicians and making such defenses illegal.
-
@joe @inthehands @ShadSterling i like to prop the demon core open with a screwdriver and point the aperture at the internet
@dotstdy You saw what happened to the last guy who did that, right?
*blue flash*
-
This is clearly how copyright law as written •should• work. Not sure if it’s how it •does• work, but if anybody’s trying, they have my sword.
What they'll *need* is your wallet.
