Open yourdomain.com/robots.txt and look for Disallow rules under AI crawler user agents, or a blanket User-agent: * Disallow: /. Allow the retrieval crawlers at minimum if you want AI shopping visibility – training crawlers are a separate decision.
The two classes that matter:
- Retrieval and search crawlers put you in AI answers: OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot and Claude-User (Anthropic), PerplexityBot, Bingbot (which also feeds ChatGPT’s organic results). Blocking these is opting out of AI shopping.
- Training crawlers feed model training: GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot, Bytespider. Blocking these is a content-licensing stance, and a legitimate one – it doesn’t remove you from AI answers today.
One that confuses people: Google-Extended is a robots.txt token, not a crawler that fetches pages. It governs whether Google uses your content for Gemini. Google’s AI Overviews ride the normal Google index via Googlebot, so blocking Google-Extended doesn’t remove you from AI Overviews – and blocking Googlebot removes you from Google entirely.
Where the file comes from in WordPress: a physical robots.txt in the site root overrides the virtual one WordPress generates. Edit the physical file over SFTP, or use your SEO plugin’s editor – Yoast → Tools → File editor, or Rank Math → General Settings → Edit robots.txt. Keep WooCommerce’s cart, checkout, and account disallows; they’re correct.
The fix is removing the offending Disallow lines, not adding Allow lines – a crawler with no rule against it crawls by default.
Two caveats: robots.txt is advisory, and it’s read after the edge – a Cloudflare or firewall block overrides anything written here (How Do I Allow AI Crawlers Through Cloudflare?). Blaze AI Discovery flags robots.txt rules per crawler but never edits the file – changes this consequential stay in your hands. Re-run the check after saving.