My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies.

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Fri, 03 Jul 2026 04:44:39 GMT

nicd@masto.ahlcode.fi — Fri, 03 Jul 2026 04:44:39 GMT

@jwildeboer Just this week a repository in my Forgejo instance was under attack. In a day, I racked up over 130k distinct IPs with fail2ban and had to abandon that approach.

I now have a simple trick that cut out practically all of the traffic, but I hesitate to share it as it's not difficult to work around… I wish we didn't have to resort to such things.

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Thu, 02 Jul 2026 13:56:25 GMT

rytmis@hachyderm.io — Thu, 02 Jul 2026 13:56:25 GMT

@Profpatsch @agturcz @alan @jwildeboer

Yep. I just read about it some weeks back and immediately tried to look for dumb TVs as an alternative. Of course, they don’t really exist as a product category any more, so the next best thing was to block those things at the router. ️

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Thu, 02 Jul 2026 13:48:06 GMT

profpatsch@mastodon.xyz — Thu, 02 Jul 2026 13:48:06 GMT

@rytmis @agturcz @alan @jwildeboer omg this is a monetization angle for TVs that is just so obvious when you consider the race to the bottom in that industry

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Wed, 01 Jul 2026 20:39:51 GMT

jwildeboer@social.wildeboer.net — Wed, 01 Jul 2026 20:39:51 GMT

@algernon A wonderful understatement. Perfect answer @b_b @grumpybozo @alan

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Wed, 01 Jul 2026 20:38:51 GMT

algernon@come-from.mad-scientist.club — Wed, 01 Jul 2026 20:38:51 GMT

@b_b @grumpybozo @alan @jwildeboer I've been using this trick (+ a few tweaks) for about a year now, with iocaine, with great success.

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Wed, 01 Jul 2026 19:21:32 GMT

b_b@mastodon.roflcopter.fr — Wed, 01 Jul 2026 19:21:32 GMT

@algernon @grumpybozo @alan @jwildeboer Did you try to use that trick ? What tool did you used and did it works well ?

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Tue, 30 Jun 2026 10:49:22 GMT

jwildeboer@social.wildeboer.net — Tue, 30 Jun 2026 10:49:22 GMT

@harry_wood The scrapers that hammer my server aren't using circumvention tactics and are, in fact, very stupid ones. What makes them hard to block is that they come from all over the world, with unique IP addresses that have no clearly identifiable origin. That's the "residential IP proxy" effect. They do a few requests and disappear again. So rate limiting doesn't catch them. @TimWardCam

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Tue, 30 Jun 2026 09:59:51 GMT

harry_wood@en.osm.town — Tue, 30 Jun 2026 09:59:51 GMT

@jwildeboer @TimWardCam Yes. This was puzzling me. Surely the big AI providers, OpenAI, Google, etc, wouldn't want to damage their brand by operating scrapers so incompetently.

But no. It's not them. The scraperpocalypse coincides with the arrival of LLMs *partly* because of increased demand for data sets, but partly just because LLM coding enables vast armies of script kiddies to easily develop scrapers that use circumvention tactics.

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Tue, 30 Jun 2026 07:04:00 GMT

dascandy@infosec.exchange — Tue, 30 Jun 2026 07:04:00 GMT

@DamonHD @nxadm @koen_hufkens @jwildeboer Not so much compromise as shipped with the device.

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Tue, 30 Jun 2026 06:57:40 GMT

dascandy@infosec.exchange — Tue, 30 Jun 2026 06:57:40 GMT

@alan @jwildeboer The residential proxy is the *industrialized* scale use of smart TVs to host a proxy for companies to use to redirect requests through, so it's actually most people's regular TVs that are attacking you.

Which also means that if you block any of them, you're cutting off actual users too. Residences have one IP, and most users don't even know they're hosting a proxy for companies to lease.

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Tue, 30 Jun 2026 02:25:07 GMT

mgd81@infosec.exchange — Tue, 30 Jun 2026 02:25:07 GMT

@jwildeboer At work, I've simply begun blocking /8's at the firewall.... it's easier and actually causes less collateral damage than one might assume at this point.

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Mon, 29 Jun 2026 22:58:04 GMT

photo55@mastodon.social — Mon, 29 Jun 2026 22:58:04 GMT

@jwildeboer @alan
I suppose one could rate limit the site's outward traffic, without reference to where it is going ...
And then have a very long list of specific addresses which get an extra rate.
And that list, which would be like a Squid access list, could be shared in some sections among quite a lot of sites.

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Mon, 29 Jun 2026 22:56:28 GMT

chrisp@cyberplace.social — Mon, 29 Jun 2026 22:56:28 GMT

@dzwiedziu @jwildeboer Web 4.0, invite only with lobste.rs style reputation system.

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Mon, 29 Jun 2026 21:48:21 GMT

woozle@toot.cat — Mon, 29 Jun 2026 21:48:21 GMT

@jwildeboer @alan

I have a proposal: BotID

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Mon, 29 Jun 2026 20:21:41 GMT

rytmis@hachyderm.io — Mon, 29 Jun 2026 20:21:41 GMT

@agturcz @alan @jwildeboer

Apparently many ”smart” TV manufacturers ship proxy SDKs from companies like Bright, and they turn the TVs into nodes in a botnet that is used for ”AI” data scraping, so the traffic comes from all over the place.

I’d guess not many consumers know about it, let alone have the technical know-how to prevent it.

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Mon, 29 Jun 2026 18:55:39 GMT

jab01701mid@mastodon.social — Mon, 29 Jun 2026 18:55:39 GMT

@jwildeboer @alan It's sad that we can't use MAC address-based filtering on the IoT client devices themselves. All of them reserve blocks of MAC addresses, usually from the NIC manufacturer's block, where it would be easy to block all traffic from "Samsung TV sets" or "Roku Devices" or "Apple TV".

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Mon, 29 Jun 2026 18:42:48 GMT

timwardcam@c.im — Mon, 29 Jun 2026 18:42:48 GMT

@dalias @rupert @jwildeboer People were doing this sort of thing before the AI garbage.

One place I worked, several years ago now, there was a performance problem.

"We'd better fix that by tweaking the cloud autoscaling parameters" they said.

FFS. I had a look at the actual code and made it go several times faster.

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Mon, 29 Jun 2026 18:39:29 GMT

algernon@come-from.mad-scientist.club — Mon, 29 Jun 2026 18:39:29 GMT

@grumpybozo @alan @jwildeboer FWIW, you can still mitigate most of them if you look at headers other than the user agent.

Many of the crawlers that try to disguise themselves as real browsers utterly fail at sending headers those browsers would, like sec-fetch-mode on any HTTPS request.

With few exceptions, if the UA contains Chrome/ or Firefox/, and the request doesn't have a sec-fetch-mode header, the chance of it being a crawler is almost certain.

I've been successfully mitigating pretty much all of them for about a year now (from ~100 million requests/day down to 3 million, the majority of which is served garbage).

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Mon, 29 Jun 2026 18:39:16 GMT

dalias@hachyderm.io — Mon, 29 Jun 2026 18:39:16 GMT

@rupert @TimWardCam @jwildeboer Exactly. This is *ideological* - they deem actually-engineered solutions that do things efficiently as a backwards "dirty human" way of doing things. Obviously since their AI slop is superior, they should do the scraping in whatever way the AI slop vomits out code to do it.

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Mon, 29 Jun 2026 18:36:57 GMT

dalias@hachyderm.io — Mon, 29 Jun 2026 18:36:57 GMT

@alan @jwildeboer Yes, it is entirely Apple's and Google's fault that they are hosting botnet malware in their "walled gardens" as legitimate and vetted software. Without that, the botnets would not exist on any viable scale.

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Mon, 29 Jun 2026 16:02:05 GMT

avuko@infosec.exchange — Mon, 29 Jun 2026 16:02:05 GMT

@alan @jwildeboer

It is all an accountability problem. ‍️

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Mon, 29 Jun 2026 15:56:06 GMT

grumpybozo@toad.social — Mon, 29 Jun 2026 15:56:06 GMT

@alan @jwildeboer The attacks have thwarted all of those tactics. They use UAs constructed from real UA tokens with minor variations. They have graduated from cheap VMs on Huawei Cloud and Digital Ocean to random IoTs in millions of households and mobile devices in millions of hands.
A few days ago I was able to measure over a thousand simultaneous sessions, each from a different /16 network.
My response to that isn’t taking the site down, but I am shedding load aggressively.

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Mon, 29 Jun 2026 14:50:58 GMT

froztbyte@mastodon.social — Mon, 29 Jun 2026 14:50:58 GMT

@LeonardoDiOttio @alan @jwildeboer it's not only that kind of stuff, fwiw - go look up bright data's sdk (for example), then do some speculative math on how many people are out there with phones that are full of apps with that sort of shit in it (there's more than only bright-sdk out there)

Reply to My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. on Mon, 29 Jun 2026 14:20:53 GMT

jwildeboer@social.wildeboer.net — Mon, 29 Jun 2026 14:20:53 GMT

@maswan Ouch