Usually rate limiting or a leftover custom block, not the malware firewall. Wordfence’s crawler rate limits throttle or block bots that request many pages quickly – which is exactly how AI crawlers behave – and blanket bot rules added by a previous developer catch the rest.
- Diagnose before changing anything: Wordfence → Tools → Live Traffic. Filter to blocked requests and find the crawler’s user agent. Each entry names the rule that fired, which tells you where to go next.
- Wordfence → Firewall → Blocking. Remove custom patterns that match AI crawler user agents (GPTBot, ClaudeBot, PerplexityBot, Bytespider – check what’s listed) and any broad datacenter IP-range blocks.
- Wordfence → All Options → Rate Limiting. The settings “If a crawler’s page views exceed” and “If anyone’s requests exceed” are the usual culprits on catalogs with hundreds of products – a crawler indexing your store trips them in minutes. Set the action to throttle rather than block, or raise the thresholds. Leave “fake Googlebot” verification on; that rule is earning its keep.
- Don’t chase IP allowlists. AI crawler IPs rotate across large ranges. Fix the rule that’s firing; allowlisting today’s addresses breaks again next week.
- Verify. Re-run the crawler check in Blaze AI Discovery. If Wordfence is clean and the block persists, the layer above you is doing it – see How Do I Allow AI Crawlers Through Cloudflare?.