AI Bot Access Checker

"Blocked in robots.txt" means your robots.txt file has a rule that tells that crawler not to fetch the page, which is a choice you can edit directly. "Blocked by server" means we sent a live request using the bot's User-Agent and your site rejected it (typically a 403, 429, or timeout) even though robots.txt allowed it. The server block almost always comes from a CDN or WAF rule, such as Cloudflare's "Block AI bots" toggle, rather than from anything on your site itself.

robots.txt is only a request, and it tells you nothing about edge or firewall rules that sit in front of your pages. A site can allow PerplexityBot in robots.txt while a CDN silently returns a 403 to that same User-Agent, so the crawler never reaches the content. These silent server blocks are a common reason sites get no AI citations despite a clean robots.txt, which is why the tool sends a real request per bot.

Blocking training crawlers like GPTBot, CCBot, and Bytespider is a legitimate choice if you do not want your content used for model training. Blocking answer-engine crawlers like OAI-SearchBot and PerplexityBot is different, because that is what removes you from ChatGPT Search and Perplexity results. These AI bots are separate from Googlebot, so allowing or blocking them does not change your normal Google Search rankings.

Some entries such as Google-Extended and Applebot-Extended are opt-out signals rather than active crawlers, so there is no live request to test against them. For those we evaluate only the robots.txt rule and label them "robots.txt only." They control whether your content can be used for AI training features, not whether a separate bot fetches your pages.