Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
-
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
@inthehands
What guarantee does one have that Google will abide by these restrictions? -
@cceckman The contract I thought I was signing was this: I published my stuff on a worldwide information network, with no controls whatever, specifically so that anyone anywhere could access it. I did that with full understanding that it would enable people I might not like to read, copy, and share it and put it to uses that I couldn't foresee and might not approve of. And if I didn't want to entertain that possibility I should not have installed a program on my computer whose sole purpose was to deliver of my stuff to any rando who asked for it.
I'm not saying I got a good deal, or that I'm happy with the outcome. But I'm not going to pretend I was tricked or that Google reneged on a bargain. We had no bargain. I served them the stuff anyway, whenever they asked for it.
And I'm not sure I believe Paul Cantrell when he says he thought the contract was different from what I said.
@mjd (thanks- this made me write down some of the thoughts I've been kicking around for a bit!)
"publish freely and for free" is also my site's situation, but it is a privileged one.
My mom's website is a funnel for her books, courses, services, etc. It is useful for her to publish helpful things for people to learn from; and to have those indexed in search; so that when people look for an answer to a question, they find out about her, and may wind up giving her money in the future.
Someone *can* scrape what she writes and re-post it, sure. And that's (in my opinion) a bad thing to do, because now the person who has done the work doesn't get any benefits from it: reputation, income, whatever. People *can* do that- SEO farms, image-sharing sites, whatnot- but as long as there is enough traffic going to the author/creator, there's still a viable business in "write for the web". (There is at least some legal consensus around this idea in the form of copyright law- that an author should be compensated for their work.)
What happens to that business model if "search" leads, not to the author's site, but to a hologram of it? Without providing the author any benefit, in citation (reputation), ad revenue, or sales funnel?
Some authors (like you, maybe? and like me?) will still write and publish. Wikipedia editors will still edit! The web doesn't fully disappear.
But...the incentives to provide that information *professionally* are degraded; the income has to come from somewhere else. And there's a lot of bad ideas that have better funding than some good ones.
If Anthropic scrapes the Crochet Answer Book (they did) and answers questions from it (it does), then who is going to write the next edition? Are Anthropic, Google, OpenAI going to pay for the one copy each they need to buy? Or is it necessarily the effort of someone who is not fairly paid for it?
If we get fully automated luxury space communism, fine, share everything for free; but, in the world we're in-- I'm worried that there's no longer a *financial* incentive to share reliable information openly. So some of those sources will dry up.
-
@joe
It is and some of us miiiiight already be doing it.@inthehands @joe gotta update my robots.txt real quick
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands IMO anything besides active denial measures (proof of work e.g. Anubis) is pointless. These companies don’t care about consent, and if they do that care is completely predicated on a few important nerds being in positions of power. That won’t last forever, especially in an environment where a larger and larger number of software engineers are people cashing a check rather than people who have a values based commitment to an open internet.
-
@wronglang @mjd @cceckman this sort of discrepancy is why I’ve never liked the term “social contract” - it’s nothing like a “contract”
@ShadSterling @mjd @cceckman yeah fair, I only commented because this is one place the distinction matters in that a social contract exists in aggregate as a set of expectations regardless of what an individual might expect or feel like they agreed to
-
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
-
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
-
@joe @inthehands is there a coordinated effort that has a website? And/or server plugins that automate serving coordinated poison?
@ShadSterling @inthehands i don't know if there's a coordinated movement. there are prefab tools like https://lib.rs/crates/iocaine that are relatively easy to deploy, though i imagine they also lose some of their effectiveness as they become more popular and LLM providers start to counter them
-
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands @iwein HEAR HEAR!!
-
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
@inthehands One of the things I've done recently is to bring enforcing robots.txt within my webserver engine. The /robots.txt itself still exists; the vast majority of it is a list of bots that are `Disallow: /` .
I still get a few of these bots attempting to hit the site, so it's definitely doing something.
-
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
@inthehands you can block their bots at the network level
-
@ShadSterling @inthehands i don't know if there's a coordinated movement. there are prefab tools like https://lib.rs/crates/iocaine that are relatively easy to deploy, though i imagine they also lose some of their effectiveness as they become more popular and LLM providers start to counter them
@joe @ShadSterling
I share Joe’s concern that poison-in-box systems will become detectable, but they seem like a good place to start.I’m even more a fan of bespoke one-off poison generators for those of us who have the means to write them. Both/and.
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands I guess “Search Engine Optimisation” has gained a new, contrary meaning.
-
@joe @ShadSterling
I share Joe’s concern that poison-in-box systems will become detectable, but they seem like a good place to start.I’m even more a fan of bespoke one-off poison generators for those of us who have the means to write them. Both/and.
@inthehands @ShadSterling if you can live with exposing yourself to the demon core a bit, using a locally hosted LLM to generate the poison also seems like a good way to get a lot of hard-to-detect variability, and hopefully also slightly accelerate model collapse in the process
-
@inthehands I guess “Search Engine Optimisation” has gained a new, contrary meaning.
“search engine nopetimization”
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands Hey, I'm curious if you have an opinion on what that contract *should* say? I don't think "pay with traffic" has worked for a long time. What would?
-
@inthehands As I said just a while ago: Every big tech press event these last few years have felt like "Announcing our exciting plans for oligarchs to strip-mine the entire world and immiserate all of humanity! Get on board, and also death to the unbelievers!"
My recommendation is always "Follow the Money" .
Google is now an adjunct of the fossil fuel industry & its fossil fuel funded public corruption.Go after the wealth of the billionaires & oil oligarchs funding Google's #Enshittification
#PrinceBonesaw
Alwaleed bin Talal
Chris Hohn
Elon Musk
Sergey Brin
Peter Thiel
Larry Ellison
Charles Kochhttps://www.washingtonpost.com/technology/2025/05/13/trump-tech-execs-riyadh/
https://www.sfgate.com/tech/article/billionaire-hohn-more-google-layoffs-17736530.php
Fossil fuel phase out.
-
This is clearly how copyright law as written •should• work. Not sure if it’s how it •does• work, but if anybody’s trying, they have my sword.
It almost certainly is but that means going to court. I've not yet seen anyone trying.
Someone who can demonstrate avtual financial damages absolutely should. Especially with the explicit statements of intent from Google now.
-
@inthehands @ShadSterling if you can live with exposing yourself to the demon core a bit, using a locally hosted LLM to generate the poison also seems like a good way to get a lot of hard-to-detect variability, and hopefully also slightly accelerate model collapse in the process
@joe @inthehands @ShadSterling i like to prop the demon core open with a screwdriver and point the aperture at the internet
-
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
@inthehands All this, and a little more! When Google *does* start ignoring robots.txt and other mechanisms, that's another victory for us, not them, even if it means we have to react to it.
Not all of Google's infrastructure is servers in a giant building, or software systems running on top of it, or even offices full of stressed out tech workers. Part of their infrastructure, the cladding on the castle walls, is their false pretense of being good citizens on the internet. When we call their bluff and they eventually drop the pretense, that's us getting them to tear down the outer layers of the castle themselves. We know what they are, and we can make them admit it, and that's power.
