In the Cloudflare dashboard, not robots.txt – an edge block fires before robots.txt is read. Set the AI bot policies to allow Search and Agent crawlers, check Bot Fight Mode, and remove any WAF rules that match AI user agents.
- Log in and select your domain. If your host manages Cloudflare for you and you have no dashboard access (Kinsta works this way), see Does Kinsta Block AI Crawlers?.
- Security → Settings → AI bot policies. Since July 1, 2026, Cloudflare sorts AI bots into three classes: Search, Agent, and Training. For AI shopping visibility, set Search and Agent to Allow. Training is a separate content-licensing decision – read What Changes in Cloudflare on September 15, 2026 – and Can It Block Googlebot? before blocking it, because of how it interacts with Googlebot.
- Security → Bots. On the free plan, Bot Fight Mode challenges automated traffic with no exceptions and can 403 verified AI crawlers. Turn it off, or on paid plans use Super Bot Fight Mode with “Verified bots” set to Allow.
- Security → WAF → Custom rules. Look for rules matching user agents (GPTBot, ClaudeBot, PerplexityBot) or blanket bot blocks a previous developer added. Delete them, or add a skip rule for verified bots.
- Check you’re not in Under Attack Mode. It challenges every visitor, crawlers included. It’s for emergencies, and some sites forget to turn it off.
- Verify from the outside. Your own server can be allowlisted, which masks the result. Re-run the crawler check in Blaze AI Discovery – the outside-in check fetches your pages as each crawler from the open internet and reports a per-crawler verdict.