My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies.
-
My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. They seem to be using botnets (or what some call „residential IP proxies“ to make it sound a bit more legitimate) with millions of IP addresses, making it really hard to defend against. Some have decided to take their sites down until this is over. This is now the world we live in
@jwildeboer Yup. Our free software mirror sometimes takes a minute to respond to an apt update, because there's millions of scraper IPs hitting the entire namespace (we use a fast caching of popular files for performance) saturating the backend storage.
We've tried various blocking and qos approaches, but we have yet to find something that really helps.
Baseline performance for us is 10-40Gbit/s, and we are now down to hardware upgrades in the hopes that real users will have enough left over.
-
@jwildeboer Why would an AI company need millions of copies of the same data?
@TimWardCam It's not necessarily the AI companies themselves. There's a whole new sector of (VC-backed) startups that claim to be able to deliver perfectly clean and curated training data for domain-specific models. And in a weird turn of events, they find out that many crawlers running in big datacenters are now being blocked by many sites they want to scrape. So using the "residential proxy IP" botnets seems to them a good option.
-
@jwildeboer Yup. Our free software mirror sometimes takes a minute to respond to an apt update, because there's millions of scraper IPs hitting the entire namespace (we use a fast caching of popular files for performance) saturating the backend storage.
We've tried various blocking and qos approaches, but we have yet to find something that really helps.
Baseline performance for us is 10-40Gbit/s, and we are now down to hardware upgrades in the hopes that real users will have enough left over.
@maswan Ouch
-
@alan @jwildeboer As these IPs are largely from people’s personal connections (for instance because they have a malware infected Smart TV or router, or run some kind of smart TV/free game/browser extension with this dubious code deliberately inserted in it) you would effectively be blocking entire consumer ISPs.
If you run a regular website you want people using regular consumer ISPs to reach it.
That makes the use of these proxies so effective.
@LeonardoDiOttio @alan @jwildeboer it's not only that kind of stuff, fwiw - go look up bright data's sdk (for example), then do some speculative math on how many people are out there with phones that are full of apps with that sort of shit in it (there's more than only bright-sdk out there)
-
@jwildeboer It's weird that the response is to take sites down rather than reach for technical countermeasures -- rate limiting, UA filtering, datacenter ASN blocks. Is the residential proxy problem genuinely that hard to solve at scale, or is the downtime itself the point? A visible protest signal rather than a quiet WAF tweak feels like a different kind of statement about where people think the leverage actually is?
@alan @jwildeboer The attacks have thwarted all of those tactics. They use UAs constructed from real UA tokens with minor variations. They have graduated from cheap VMs on Huawei Cloud and Digital Ocean to random IoTs in millions of households and mobile devices in millions of hands.
A few days ago I was able to measure over a thousand simultaneous sessions, each from a different /16 network.
My response to that isn’t taking the site down, but I am shedding load aggressively. -
@jwildeboer 2/2
The deeper problem is upstream. Apple, Google and Microsoft are allowing SDK-injected bandwidth harvesting through their app stores. Until that's addressed at source, we're all playing whack-a-mole with an essentially infinite residential IP pool. This isn't a mail security problem -- it's a platform accountability problem.It is all an accountability problem.
️ -
@jwildeboer 2/2
The deeper problem is upstream. Apple, Google and Microsoft are allowing SDK-injected bandwidth harvesting through their app stores. Until that's addressed at source, we're all playing whack-a-mole with an essentially infinite residential IP pool. This isn't a mail security problem -- it's a platform accountability problem.@alan @jwildeboer Yes, it is entirely Apple's and Google's fault that they are hosting botnet malware in their "walled gardens" as legitimate and vetted software. Without that, the botnets would not exist on any viable scale.
-
@TimWardCam @jwildeboer Because their trawler is as sloppily coded as everything else they do.
@rupert @TimWardCam @jwildeboer Exactly. This is *ideological* - they deem actually-engineered solutions that do things efficiently as a backwards "dirty human" way of doing things. Obviously since their AI slop is superior, they should do the scraping in whatever way the AI slop vomits out code to do it.
-
@alan @jwildeboer The attacks have thwarted all of those tactics. They use UAs constructed from real UA tokens with minor variations. They have graduated from cheap VMs on Huawei Cloud and Digital Ocean to random IoTs in millions of households and mobile devices in millions of hands.
A few days ago I was able to measure over a thousand simultaneous sessions, each from a different /16 network.
My response to that isn’t taking the site down, but I am shedding load aggressively.@grumpybozo @alan @jwildeboer FWIW, you can still mitigate most of them if you look at headers other than the user agent.
Many of the crawlers that try to disguise themselves as real browsers utterly fail at sending headers those browsers would, like
sec-fetch-modeon any HTTPS request.With few exceptions, if the UA contains
Chrome/orFirefox/, and the request doesn't have asec-fetch-modeheader, the chance of it being a crawler is almost certain.I've been successfully mitigating pretty much all of them for about a year now (from ~100 million requests/day down to 3 million, the majority of which is served garbage).
-
@rupert @TimWardCam @jwildeboer Exactly. This is *ideological* - they deem actually-engineered solutions that do things efficiently as a backwards "dirty human" way of doing things. Obviously since their AI slop is superior, they should do the scraping in whatever way the AI slop vomits out code to do it.
@dalias @rupert @jwildeboer People were doing this sort of thing before the AI garbage.
One place I worked, several years ago now, there was a performance problem.
"We'd better fix that by tweaking the cloud autoscaling parameters" they said.
FFS. I had a look at the actual code and made it go several times faster.
-
@alan These botnets are more or less immune to rate limiting, as they use many (and I mean millions) of IP addresses fro a run and each IP address is only used for a few requests before it is being put back in the queue. The IP addresses are also from many different providers, so a (sub-)net wide block also doesn't help. I wrote about those "residential IP proxies in [1] and [2].
[1] https://jan.wildeboer.net/2025/02/Blocking-Stealthy-Botnets/
[2] https://jan.wildeboer.net/2025/04/Web-is-Broken-Botnet-Part-2/@jwildeboer @alan It's sad that we can't use MAC address-based filtering on the IoT client devices themselves. All of them reserve blocks of MAC addresses, usually from the NIC manufacturer's block, where it would be easy to block all traffic from "Samsung TV sets" or "Roku Devices" or "Apple TV".
-
@alan I'm sorry for asking that, but do you actually know what "residential proxy" is?
Apparently many ”smart” TV manufacturers ship proxy SDKs from companies like Bright, and they turn the TVs into nodes in a botnet that is used for ”AI” data scraping, so the traffic comes from all over the place.
I’d guess not many consumers know about it, let alone have the technical know-how to prevent it.
-
@alan These botnets are more or less immune to rate limiting, as they use many (and I mean millions) of IP addresses fro a run and each IP address is only used for a few requests before it is being put back in the queue. The IP addresses are also from many different providers, so a (sub-)net wide block also doesn't help. I wrote about those "residential IP proxies in [1] and [2].
[1] https://jan.wildeboer.net/2025/02/Blocking-Stealthy-Botnets/
[2] https://jan.wildeboer.net/2025/04/Web-is-Broken-Botnet-Part-2/I have a proposal: BotID
-
@jwildeboer
This is going to end up with invitation-only graynet, where you'll be banned the moment you'll try trawling.@dzwiedziu @jwildeboer Web 4.0, invite only with lobste.rs style reputation system.
-
@alan These botnets are more or less immune to rate limiting, as they use many (and I mean millions) of IP addresses fro a run and each IP address is only used for a few requests before it is being put back in the queue. The IP addresses are also from many different providers, so a (sub-)net wide block also doesn't help. I wrote about those "residential IP proxies in [1] and [2].
[1] https://jan.wildeboer.net/2025/02/Blocking-Stealthy-Botnets/
[2] https://jan.wildeboer.net/2025/04/Web-is-Broken-Botnet-Part-2/@jwildeboer @alan
I suppose one could rate limit the site's outward traffic, without reference to where it is going ...
And then have a very long list of specific addresses which get an extra rate.
And that list, which would be like a Squid access list, could be shared in some sections among quite a lot of sites. -
My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. They seem to be using botnets (or what some call „residential IP proxies“ to make it sound a bit more legitimate) with millions of IP addresses, making it really hard to defend against. Some have decided to take their sites down until this is over. This is now the world we live in
@jwildeboer At work, I've simply begun blocking /8's at the firewall.... it's easier and actually causes less collateral damage than one might assume at this point.
-
@jwildeboer It's weird that the response is to take sites down rather than reach for technical countermeasures -- rate limiting, UA filtering, datacenter ASN blocks. Is the residential proxy problem genuinely that hard to solve at scale, or is the downtime itself the point? A visible protest signal rather than a quiet WAF tweak feels like a different kind of statement about where people think the leverage actually is?
@alan @jwildeboer The residential proxy is the *industrialized* scale use of smart TVs to host a proxy for companies to use to redirect requests through, so it's actually most people's regular TVs that are attacking you.
Which also means that if you block any of them, you're cutting off actual users too. Residences have one IP, and most users don't even know they're hosting a proxy for companies to lease.
-
@nxadm @koen_hufkens @jwildeboer One set of those 'residential proxies' is apparently compromised 'smart' TVs; another is stuff silently embedded in 'free' mobile games. We all pay the price for those.
@DamonHD @nxadm @koen_hufkens @jwildeboer Not so much compromise as shipped with the device.
-
@TimWardCam It's not necessarily the AI companies themselves. There's a whole new sector of (VC-backed) startups that claim to be able to deliver perfectly clean and curated training data for domain-specific models. And in a weird turn of events, they find out that many crawlers running in big datacenters are now being blocked by many sites they want to scrape. So using the "residential proxy IP" botnets seems to them a good option.
@jwildeboer @TimWardCam Yes. This was puzzling me. Surely the big AI providers, OpenAI, Google, etc, wouldn't want to damage their brand by operating scrapers so incompetently.
But no. It's not them. The scraperpocalypse coincides with the arrival of LLMs *partly* because of increased demand for data sets, but partly just because LLM coding enables vast armies of script kiddies to easily develop scrapers that use circumvention tactics.
-
@jwildeboer @TimWardCam Yes. This was puzzling me. Surely the big AI providers, OpenAI, Google, etc, wouldn't want to damage their brand by operating scrapers so incompetently.
But no. It's not them. The scraperpocalypse coincides with the arrival of LLMs *partly* because of increased demand for data sets, but partly just because LLM coding enables vast armies of script kiddies to easily develop scrapers that use circumvention tactics.
@harry_wood The scrapers that hammer my server aren't using circumvention tactics and are, in fact, very stupid ones. What makes them hard to block is that they come from all over the world, with unique IP addresses that have no clearly identifiable origin. That's the "residential IP proxy" effect. They do a few requests and disappear again. So rate limiting doesn't catch them. @TimWardCam
