Our web crawler
SEOAIAuditsBot is the crawler behind SEO AI Audits. When someone runs a free SEO audit, it fetches the public pages reachable from the URL they submitted, reads the HTML, and checks them against our audit modules. It is not a scraper — it only visits pages for an audit or a scheduled check that someone asked for, obeys robots.txt unless the site’s owner has verified the domain with us, crawls at most 600 pages per site, and never touches anything behind a login.
How to identify it
Every request our auditor makes carries these headers so you can recognise and allow it:
X-SEOAIAudits-Bot: https://seoaiaudits.com/bot From: [email protected]
Our published crawler User-Agent is Mozilla/5.0 (compatible; SEOAIAuditsBot/1.0; +https://seoaiaudits.com/bot). For maximum compatibility the live audit crawl may present a standard browser User-Agent — so the most reliable ways to recognise us are the X-SEOAIAudits-Bot header above and the cryptographic request signature described below.
The same server also runs TechShuBot, a separate crawler with its own name and rules. See TechShuBot.
Verifying our identity
SEOAIAuditsBot identifies itself cryptographically, so you can confirm a request genuinely comes from us — not someone spoofing our name:
- Signed requests (Web Bot Auth). Every request carries an Ed25519 HTTP Message Signature (RFC 9421 / Cloudflare Web Bot Auth). Verify it against our published public-key directory:
https://seoaiaudits.com/.well-known/http-message-signatures-directory
Allowlist it to audit your own site
If your site is behind a firewall or CDN (Cloudflare, Vercel, AWS WAF, etc.), bot protection may block our crawler and your report will be incomplete. To audit a site you own, add an allow rule for our identifying header:
Cloudflare
WAF → Custom rules → create a Skip rule with this expression:
(http.request.headers["x-seoaiaudits-bot"][0] eq "https://seoaiaudits.com/bot")
Vercel
Firewall → add a rule that Bypasses the challenge when the request header X-SEOAIAudits-Bot equals https://seoaiaudits.com/bot.
Other firewalls / by IP
Most WAFs support an allow/bypass rule on a request header. If you can only allow by IP address, contact us for our current source IP.
These rules match a request header, and anyone can send a header, so a rule like this also lets through any request that copies it. Scope it as narrowly as your firewall allows, and remove it when you no longer need audits from behind the firewall.
How it behaves
- Obeys robots.txt — skips any path disallowed for our user-agent or the * group. While a domain carries our verification record, full audits of it also crawl the paths its robots.txt disallows (see Verify your site).
- Only fetches public pages reachable from a URL a user explicitly submitted for an audit.
- Crawls at most 600 pages per site, paces its requests, and automatically backs off when your server signals load (HTTP 429 / 503).
- Reads only publicly served HTML — never anything behind a login, and never submits forms.
- Cryptographically signs and identifies every request (see above) instead of hiding.
- Does not resell or republish the text of your pages. An audit's results go to the person who ran it and to anyone they share the report link with, and a domain's latest scores can be looked up on our comparison pages.
Questions or want it to stop?
If you have questions about SEOAIAuditsBot, want to confirm a request really came from us, or want us to stop crawling a site, email [email protected]. You can also run a free audit or browse our free SEO tools.