I’m scraping a website using Selenium. I’m only clicking a few times and I’m not gathering a huge amount of data yet. There's no captcha or anything either.
Sometimes my code works just fine but other times the website denies my request, displaying the following text:
The requested URL was rejected. Please consult with your administrator.
Whenever I tried manually, I’ve never received this message. However I see so many options for how to avoid bot detection. What would be the most likely solution?
I’ve spoofed my user-agent, had it use incognito mode, added random waits (explicit waits if that makes a difference), waited for things to become visible before clicking, and cleared my cache for the website. I’ve been asking GPT and reading blogs for common bot detection techniques and how to avoid detection but
None of them really seem to prevent this message. The message occurs occasionally and persists when it does.
