Free AI Crawler Log Analyzer

AI crawlers identify themselves in the User-Agent header of every request — for example GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot. Your web server records every request (with its User-Agent and IP) in an access log. This tool reads that log and surfaces exactly which AI bots hit your site, how many times, and which pages they fetched.

Google Analytics runs as a client-side JavaScript tag. OpenAI’s and Anthropic’s crawlers, and Perplexity’s, do not run JavaScript, so it never fires for them — they are invisible to it. The only reliable way to detect bot visits is server-side: your access logs (or your CDN/Cloudflare logs), which is exactly what this tool analyzes.

Three fetchers run only when a person asks an assistant a question, rather than on a crawl schedule: ChatGPT-User, Claude-User and Perplexity-User. OpenAI documents ChatGPT-User visiting a web page when “users ask ChatGPT or a CustomGPT a question” (developers.openai.com/api/docs/bots, read 2026-09-23), and Perplexity says Perplexity-User fetches a page “when users ask Perplexity a question” (docs.perplexity.ai/guides/bots, read 2026-09-23). Seeing those hits per page is the closest a server log gets to evidence that your page was read while an answer was being written. It is not proof of a citation: an assistant can read a page and never link it.

Yes, and they are reported separately from crawler hits, because being crawled and being visited are different events that must never be added together. The tool counts requests whose referrer is chatgpt.com, perplexity.ai, gemini.google.com, copilot.microsoft.com or claude.ai, plus landing URLs carrying utm_source=chatgpt.com. Treat the result as a minimum: some assistants and in-app browsers send no referrer at all. Google’s AI Overviews and AI Mode cannot be separated this way — Google Analytics 4 draws the same line, defining its AI Assistant channel as arrivals “from sources like ChatGPT, Gemini, Deepseek, Copilot, or Grok” while excluding AI Overviews and AI Mode (support.google.com/analytics/answer/9756891, read 2026-09-23).

Yes. Cloudflare Logpush writes one JSON record per line by default — “each record as a single line of JSON (also known as ndjson)” (developers.cloudflare.com/logs/logpush/logpush-job/log-output-options/, read 2026-09-23) — and this tool reads that format as well as the Combined Log Format nginx and Apache write. It uses Cloudflare’s own field names, so ClientRequestUserAgent, ClientRequestReferer, ClientRequestURI, ClientIP and EdgeStartTimestamp all come through. The tool tells you which format it read your file as, so a mis-detected export can never look like a quiet week.

Anyone can send a fake User-Agent claiming to be GPTBot. To catch impostors, this tool checks each request’s IP against the range file the operator publishes for that bot. We cover OpenAI’s (GPTBot, OAI-SearchBot, ChatGPT-User), Perplexity’s (PerplexityBot, Perplexity-User) and Anthropic’s (ClaudeBot, Claude-User, Claude-SearchBot): our server re-fetches those files from the operators every day, and the page downloads that copy when it loads, falling back to the snapshot built into the page if the download fails. Everything else shows as “unverified” — that means we did not check, not that the bot is suspect. Google and Microsoft publish files too, but a miss against either cannot support an accusation: Google says its crawlers “generally” crawl from its published ranges (developers.google.com/search/docs/crawling-indexing/google-common-crawlers, read 2026-09-23), and Microsoft’s bingbot.json was stamped January 2024 when we read it on 2026-09-23. Both can still be verified by reverse DNS, which a browser-only tool cannot perform. Whichever copy the tool ends up with, it is used for 30 days and no longer: if our refresh stops, verification switches itself off and everything reads “unverified”, because these ranges churn — about half of OpenAI’s ChatGPT-User prefixes turned over in the 93 days before our 2026-09-23 snapshot — and a stale copy would accuse genuine traffic of being fake. Two innocent explanations exist for an out-of-range hit even so: the operator may have added addresses since our last refresh, or your log may record a proxy’s address rather than the caller’s.

No. The entire analysis runs in your browser using JavaScript — your log file (which may contain visitor IPs) never leaves your device and is never sent to any server. The page does fetch one thing from us, the public list of crawler IP ranges, and that is a download only: no part of your log, and no count taken from it, is ever sent back. No visitor IP is displayed or stored: addresses are used only to compare against a crawler’s published range, and the referral report keeps the landing path and the assistant, never the visitor.

Most hosts expose it in cPanel/Plesk under “Raw Access Logs,” or at a path like /var/log/nginx/access.log on a VPS. If you’re on Cloudflare, export request logs with Logpush and paste or upload the resulting file — its one-JSON-record-per-line format is read directly. Either way, include the referrer field if your log format is configurable: without it, visits from AI assistants cannot be seen at all.

A search crawler that never fetches your pages cannot surface them, so it is worth chasing. Check the plain causes in order: how long a window your log actually covers, then your robots.txt (our AI Bot Access Checker reads it per crawler), then any firewall or CDN bot rule — that last one is invisible in robots.txt, and Cloudflare’s free AI Crawl Control shows it for Cloudflare sites. Crawl frequency also varies a lot; absence from one week of logs is not proof of a block.