AI Crawler Access Checker
Many firms accidentally block the AI bots they most want to be cited by, often via a security plugin or Cloudflare default. Paste your robots.txt below to see, line by line, which engines can read your site. Runs entirely in your browser.
Recommended robots.txt
A safe starting point that allows every major AI and search crawler. Replace the sitemap URL with your own.
# Allow AI search crawlers
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
Allow: /
# Standard search engines
User-agent: Googlebot
User-agent: Bingbot
Allow: /
User-agent: *
Allow: /
Sitemap: https://yourfirm.com/sitemap.xmlWhy blocked crawlers quietly kill AI visibility
If an AI engine cannot crawl your site, it cannot cite your firm, full stop. Yet blocking happens by accident all the time: a security plugin, an over-eager Cloudflare setting, or a leftover rule from the 2023 wave of firms blocking AI scrapers. The damage is invisible until you check, because nothing on your site looks broken.
The crawlers that matter most are GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, and Bingbot, since ChatGPT Search leans heavily on Bing’s index. Allow them all unless you have a deliberate reason not to. For the full context, see is your site blocking AI crawlers and 7 reasons your firm doesn’t show up in AI answers.
Frequently asked questions
Where do I find my robots.txt?
It lives at yourfirm.com/robots.txt. Open that URL in a browser and copy the whole file into the box above.
Is it safe to allow AI crawlers?
For service businesses that want to be recommended, yes. The trade-off debate is largely over: you want to be in the answers, which means letting the engines read you.
I found a blocked bot. What now?
Update robots.txt to allow it (use the recommended file above as a starting point), redeploy, then confirm the page is indexed in Google Search Console and Bing Webmaster Tools.