Finally managed to kill the traffic:
1. renamed the worker - the bot was bypassing the route on the domain and using the worker.dev domain
2. add WAF rules to block gptbot in user agent
3. serve zip bomb on request from GPT.
interesting that GPT was accessing the worker directly to bypass the WAF rule son Cloudflare
Interesting. workers.dev domains can be a liability sometimes -- if you've mapped the worker to a real zone, then you probably don't want the workers.dev zone anymore.
For what it's worth, you can disable the workers.dev zone by putting `"workers_dev": false,` in wranlger.jsonc. You can also enable Cloudflare Access on your workers.dev zone to require login (there's a switch for this in the cloudflare dashboard UI for the worker).
But of course you have to remember to do those things... I wonder if we (Cloudflare) should be more proactive in suggesting disabling/locking down the workers.dev zone once a worker is mapped to another zone...
Finally managed to kill the traffic: 1. renamed the worker - the bot was bypassing the route on the domain and using the worker.dev domain 2. add WAF rules to block gptbot in user agent 3. serve zip bomb on request from GPT.
interesting that GPT was accessing the worker directly to bypass the WAF rule son Cloudflare
Interesting. workers.dev domains can be a liability sometimes -- if you've mapped the worker to a real zone, then you probably don't want the workers.dev zone anymore.
For what it's worth, you can disable the workers.dev zone by putting `"workers_dev": false,` in wranlger.jsonc. You can also enable Cloudflare Access on your workers.dev zone to require login (there's a switch for this in the cloudflare dashboard UI for the worker).
But of course you have to remember to do those things... I wonder if we (Cloudflare) should be more proactive in suggesting disabling/locking down the workers.dev zone once a worker is mapped to another zone...
update: the bot is back now with a vengeance, sending request at about 1 request per second. ignoring robots.txt and the status code 403
serving a zip bomb and after 10 minutes, the traffic from the gpt bot disappeared..
will be serving a zip bomb to the bot to see if they stay away from my proxy