The author seems forget that AI crawlers use millions residental proxies, so simply having proof of work indeed blocks them. If they reused proof of work with different IP, they'd be detected very quickly. Or the proof of work is linked to IP. The anubis is activately updated so im not fully following it. If they re-use IP then they'd be ratelimited.
Another advantage is requiring cookie storage, which lots crawler seem be lacking for some reason. Since most common http clients support them. When the crawlers are doing lots of requests at once, needing JS engine would add a lot of overhead aswell. So its multiple factors which makes this viable method so far.
The author seems forget that AI crawlers use millions residental proxies, so simply having proof of work indeed blocks them. If they reused proof of work with different IP, they'd be detected very quickly. Or the proof of work is linked to IP. The anubis is activately updated so im not fully following it. If they re-use IP then they'd be ratelimited.
Another advantage is requiring cookie storage, which lots crawler seem be lacking for some reason. Since most common http clients support them. When the crawlers are doing lots of requests at once, needing JS engine would add a lot of overhead aswell. So its multiple factors which makes this viable method so far.
I think Anubis is meant to stop dumb automated hyperscale scraping bots, not smart singular human-driven AI agents.
[dead]