Our web crawler
SEOAIAuditsBot is the crawler behind SEO AI Audits. When someone runs a free SEO audit, it fetches the public pages reachable from the URL they submitted, reads the HTML, and checks them against our audit modules. It is not a scraper — it only visits pages for an on-demand audit, crawls at most 500 pages per site, and never touches anything behind a login.
How to identify it
Every request our auditor makes carries these headers so you can recognise and allow it:
X-SEOAIAudits-Bot: https://seoaiaudits.com/bot From: [email protected]
Our published crawler User-Agent is Mozilla/5.0 (compatible; SEOAIAuditsBot/1.0; +https://seoaiaudits.com/bot). For maximum compatibility the live audit crawl may present a standard browser User-Agent, so the most reliable way to allowlist us is the X-SEOAIAudits-Bot request header above.
Allowlist it to audit your own site
If your site is behind a firewall or CDN (Cloudflare, Vercel, AWS WAF, etc.), bot protection may block our crawler and your report will be incomplete. To audit a site you own, add an allow rule for our identifying header:
Cloudflare
WAF → Custom rules → create a Skip rule with this expression:
(http.request.headers["x-seoaiaudits-bot"][0] eq "https://seoaiaudits.com/bot")
Vercel
Firewall → add a rule that Bypasses the challenge when the request header X-SEOAIAudits-Bot equals https://seoaiaudits.com/bot.
Other firewalls / by IP
Most WAFs support an allow/bypass rule on a request header. If you can only allow by IP address, contact us for our current source IP.
How it behaves
- Only fetches public pages reachable from a URL a user explicitly submitted for an audit.
- Crawls at most 500 pages per site and paces its requests to stay light on your server.
- Reads only publicly served HTML — never anything behind a login, and never submits forms.
- Identifies itself on every request (see the headers above) instead of hiding.
- Does not resell, republish or index your content — results are shown only to the person who ran the audit.
Questions or want it to stop?
If you have questions about SEOAIAuditsBot, want to confirm a request really came from us, or want us to stop crawling a site, email [email protected]. You can also run a free audit or browse our free SEO tools.