Pleased to share a page and explainer for the AI tarpit project Science is Poetry, with legal statement, rationale(s), and a few deployment notes:
-
@JulianOliver ok. So what do I do? Just point them at those IPs?
If they all share the same IP, won’t that make them easy to block?
@narthur For now they would all share the same IP.
Both domain and IP are naturally able to be filtered at the crawler end, but as numerous sites can be hosted behind one IP, it is my belief that they will drop a domain first.
Further, it's my hope to have instances of the project running on other dedicated hosts down the road.
-
Do you have an unused domain that you would be happy to donate to a counter-offensive against unchecked & unregulated AI crawlers that scrape human-made content to simulate & deceive for profit?
If so, pls reply to this post. Your domain would become an entrypoint to the AI tarpit & Poison-as-a-Service project below, allowing concerned public to choose to use it on their sites, helping make the project more resilient to blacklisting.
@JulianOliver im down. What's the first step?
-
@lightweight Thanks again Dave, I'm going to have some homework today adding all these beaut new domains
-
@JulianOliver im down. What's the first step?
Thanks for joining in!
If you could just add these two records for the domain you wish to gift, and then let me know the domain (either DM or public) once done
A: 95.216.76.85
AAAA: 2a01:4f9:2b:c83::2 -
Do you have an unused domain that you would be happy to donate to a counter-offensive against unchecked & unregulated AI crawlers that scrape human-made content to simulate & deceive for profit?
If so, pls reply to this post. Your domain would become an entrypoint to the AI tarpit & Poison-as-a-Service project below, allowing concerned public to choose to use it on their sites, helping make the project more resilient to blacklisting.
A bit over half a million page reads a day by crawlers rn. Just to say the server is doing some good work.
-
Do you have an unused domain that you would be happy to donate to a counter-offensive against unchecked & unregulated AI crawlers that scrape human-made content to simulate & deceive for profit?
If so, pls reply to this post. Your domain would become an entrypoint to the AI tarpit & Poison-as-a-Service project below, allowing concerned public to choose to use it on their sites, helping make the project more resilient to blacklisting.
@JulianOliver looking through the replies I'll happily point a subdomain to the provided A-Record my old e-mail address domain is just barely used for the few accounts which don't allow me to change the address.
-
Do you have an unused domain that you would be happy to donate to a counter-offensive against unchecked & unregulated AI crawlers that scrape human-made content to simulate & deceive for profit?
If so, pls reply to this post. Your domain would become an entrypoint to the AI tarpit & Poison-as-a-Service project below, allowing concerned public to choose to use it on their sites, helping make the project more resilient to blacklisting.
Hi @JulianOliver,
pick subdomains of mro1.de and tell me where to point the DNS.Edit: just saw the DNS records above, I will set up some and follow up here later today.
-
Hi @JulianOliver,
pick subdomains of mro1.de and tell me where to point the DNS.Edit: just saw the DNS records above, I will set up some and follow up here later today.
-
@narthur For now they would all share the same IP.
Both domain and IP are naturally able to be filtered at the crawler end, but as numerous sites can be hosted behind one IP, it is my belief that they will drop a domain first.
Further, it's my hope to have instances of the project running on other dedicated hosts down the road.
@JulianOliver @narthur It seems to me that it would be fairly easy for them to consider that the content is junk and to treat accordingly everything coming from the same IPs. They probably keep a log of where the content used to train the LLMs is coming from (maybe with some kind of hash / pseudonymous maneer), and likely have ways to reject content from the same server IF they detect a problem from several domains linked to this server. IMO a bunch of reverse proxies / various IPs could help : they might be dumb and make it easy to polute their dataset, but probably aren't.
-
Do you have an unused domain that you would be happy to donate to a counter-offensive against unchecked & unregulated AI crawlers that scrape human-made content to simulate & deceive for profit?
If so, pls reply to this post. Your domain would become an entrypoint to the AI tarpit & Poison-as-a-Service project below, allowing concerned public to choose to use it on their sites, helping make the project more resilient to blacklisting.
@JulianOliver i have some unused. I'll let them point at A/AAAA you already mentioned. Do you need to know the exact domains?
-
@JulianOliver i have some unused. I'll let them point at A/AAAA you already mentioned. Do you need to know the exact domains?
@themadhatter Thank you! Yes knowing the domains would be needed. If you don't want the world to know they are yours, you can DM.
-
@themadhatter Thank you! Yes knowing the domains would be needed. If you don't want the world to know they are yours, you can DM.
@JulianOliver will send you a send you a DM when ready.
-
Do you have an unused domain that you would be happy to donate to a counter-offensive against unchecked & unregulated AI crawlers that scrape human-made content to simulate & deceive for profit?
If so, pls reply to this post. Your domain would become an entrypoint to the AI tarpit & Poison-as-a-Service project below, allowing concerned public to choose to use it on their sites, helping make the project more resilient to blacklisting.
@JulianOliver I set one .fi domain for this which was unused I was supposed to delete it already.
-
A bit over half a million page reads a day by crawlers rn. Just to say the server is doing some good work.
Thanks all for the fine domains! I've decided to spin up a new VM and do all the site configs and TLS chain for them at once - more efficient, less prone to error. I will get onto that on my tomorrow and report back here.
-
Pleased to share a page and explainer for the AI tarpit project Science is Poetry, with legal statement, rationale(s), and a few deployment notes:
https://julianoliver.com/projects/science-is-poetry/
The page may grow a bit. Just wanted to get it out the door.
@JulianOliver also
the web design. gotta serve the bots something nice. -
Hi @JulianOliver,
carrot.mro1.de and
car.rot.mro1.de and
ca.rr.ot.mro1.deThe pleasure is all mine.
-
Hi @JulianOliver,
carrot.mro1.de and
car.rot.mro1.de and
ca.rr.ot.mro1.deThe pleasure is all mine.
@mro Amazing, thank you! I'll have an update tomorrow once all setup.
-
Do you have an unused domain that you would be happy to donate to a counter-offensive against unchecked & unregulated AI crawlers that scrape human-made content to simulate & deceive for profit?
If so, pls reply to this post. Your domain would become an entrypoint to the AI tarpit & Poison-as-a-Service project below, allowing concerned public to choose to use it on their sites, helping make the project more resilient to blacklisting.
@JulianOliver sign me up
-
@JulianOliver I set one .fi domain for this which was unused I was supposed to delete it already.
@korkeala Thank you! Please share via DM so I can add it to the list.
-
@JulianOliver @narthur It seems to me that it would be fairly easy for them to consider that the content is junk and to treat accordingly everything coming from the same IPs. They probably keep a log of where the content used to train the LLMs is coming from (maybe with some kind of hash / pseudonymous maneer), and likely have ways to reject content from the same server IF they detect a problem from several domains linked to this server. IMO a bunch of reverse proxies / various IPs could help : they might be dumb and make it easy to polute their dataset, but probably aren't.
I am looking to populate the tarpit to other hosts, but for now the bots just keep chewing, and have been for days at one end-point.
I suspect there are so many crawlers spawned, and that they have so much in the way of resources at hand to do this scraping, that it is largely automated with little oversight.
