How to Reduce a High Crawl-Delay in Robots.txt
A Crawl-delay directive in robots.txt tells crawlers to wait a set number of seconds between requests, throttling how fast your site gets discovered and reindexed. Googlebot ignores it entirely, but Bing, Yandex, and some other bots honor it, so a leftover high value can quietly slow reindexing for no real benefit. Lower or remove it unless your server genuinely can't handle concurrent bot traffic, and use platform-specific tools for finer control instead.
What this means
Crawl-delay is a line in robots.txt, for example Crawl-delay: 10, that asks a crawler to wait that many seconds between successive requests to your site. It was never part of the official Robots Exclusion Protocol, formalized as RFC 9309 in 2022; it's a legacy convention some search engines chose to support and others never adopted, and there's no single standard for what the number means across different bots.
This audit fires when it finds a Crawl-delay value high enough to meaningfully slow discovery of new or updated pages. That's typically a leftover from an old hosting recommendation, a cautious default set years ago when the server was weaker, or a template value copied from another site without checking whether it still makes sense.
The honest complication is that "high crawl delay" doesn't mean the same thing to every bot. Googlebot, by far the most important crawler for most sites, does not support the Crawl-delay directive at all and ignores it completely. Bingbot and Yandex's crawler do honor it. Behavior for smaller search bots, monitoring tools, and AI crawlers is inconsistent and largely undocumented, so treat this setting as a Bing/Yandex-focused lever, not a universal throttle.
Why it matters
For Google, this setting is close to irrelevant. Google's own documentation states Googlebot doesn't support Crawl-delay, and that it determines crawl rate automatically based on how a server responds: answer quickly and cleanly and Google will generally crawl more, return 5xx errors or slow responses and it backs off on its own. If you genuinely need to slow Googlebot down in an emergency, Google's documented approach is to temporarily return 500, 503, or 429 status codes, not to add a robots.txt directive.
For Bing and Yandex, though, a high Crawl-delay has a real, measurable cost. Crawl-delay: 10 limits a compliant bot to at most six requests a minute, roughly 360 an hour. On a site with a few thousand URLs, that means the crawler can't get through your content in a single session, and full reindexing after a big content update or migration can stretch out over days, not because Bing doesn't want to crawl you, but because your own robots.txt told it to slow down.
AI crawlers are the least certain part of this picture. Some may use generic robots.txt parsing libraries that happen to respect Crawl-delay by convention; others ignore it outright since it was never part of the formal standard they're built against. Don't assume a Crawl-delay value is managing AI-bot load in any predictable way. If load from a specific AI crawler is the actual problem, a targeted rule for that user-agent, or server-level rate limiting, is a more reliable lever than a blanket delay every bot interprets differently.
There's one legitimate reason to keep some delay in place: a genuinely under-provisioned server, cheap shared hosting with no caching layer, can be knocked over by simultaneous requests from several search and SEO crawlers at once. In that specific case, a moderate Crawl-delay is a reasonable stopgap while you fix the underlying capacity issue, not a permanent solution.
How to fix it
- 1
Check whether the delay is actually solving a real problem
Look at your hosting control panel or server logs for evidence bots are actually straining your server: CPU spikes correlated with crawler user-agents, or slow response times during known crawl windows. If you can't find that evidence, the Crawl-delay value is probably a leftover default rather than an active fix for anything.
- 2
Lower or remove the Crawl-delay line
If your hosting is reasonably modern, a VPS, a managed host, or anything sitting behind a CDN or page cache, remove the
Crawl-delayline from robots.txt entirely, or drop it to a low value like 1 if you want to keep a small buffer. This changes nothing for Google either way, and it lets Bing and Yandex crawl at their own judgment of what your server can handle. - 3
Use Bing Webmaster Tools for granular control instead of a blanket number
If you specifically need to manage Bing's crawl rate, Bing Webmaster Tools has a Crawl Control feature that lets you set a crawl-rate pattern by hour of day, so you can slow Bingbot during peak traffic hours and let it crawl faster overnight. That's more useful than one flat Crawl-delay value applied around the clock.
- 4
Fix real server strain at the infrastructure layer, not in robots.txt
If bots are genuinely overloading your server, the durable fix is more capacity or caching, a page cache or CDN in front of dynamic pages, upgraded hosting, or fixing an inefficient query that makes every crawled page expensive to render, not a permanent crawl throttle. If one specific bot, a scraper, an aggressive SEO tool, or a misbehaving AI crawler, is the actual culprit, rate-limit or block that user-agent specifically at the server or CDN level instead of slowing down every legitimate crawler along with it.
- 5
Re-check and monitor
After changing robots.txt, confirm it still returns a clean 200 (a broken robots.txt is a separate, more serious issue). Watch Bing Webmaster Tools' crawl stats and your server's response-time metrics over the following weeks to confirm crawl activity increases without degrading performance for real visitors, and adjust from there.
Example
# BEFORE - blanket, high delay every bot interprets differently (Google ignores it anyway)
User-agent: *
Crawl-delay: 20
Disallow:
# AFTER - low or removed; real server protection handled at the CDN/server level instead
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlDropping a high blanket Crawl-delay does nothing to Google, which never read it, but lets Bing and Yandex crawl and reindex the site far faster.
Platform-specific steps
Go to Configure My Site > Crawl Control and choose a crawl pattern, or a custom hourly schedule, instead of relying on a flat robots.txt Crawl-delay. This lets you slow Bingbot during business hours and let it crawl faster overnight, which is more useful than one number applied all day.
If Yoast or Rank Math manages your robots.txt (Yoast: Tools > File editor; Rank Math: General Settings > Edit robots.txt), check there for a leftover Crawl-delay line, since these plugins serve a virtual file that overrides any physical robots.txt in your web root. Remove or lower it there rather than editing a static file that isn't actually being served.
Instead of throttling every crawler with Crawl-delay, use Cloudflare's rate limiting or Bot Management (on eligible plans) to target only the specific bots or IP ranges causing real load, leaving Googlebot, Bingbot, and other well-behaved crawlers unaffected. This solves the server-strain problem Crawl-delay was a blunt attempt at, without slowing down legitimate discovery.
If particular crawlers are hammering your server, apply limit_req (Nginx) or mod_ratelimit / mod_evasive (Apache) scoped to specific user-agents or IP ranges, rather than a blanket robots.txt delay every bot interprets differently. This protects the server directly regardless of whether a given crawler respects Crawl-delay at all.
Frequently asked
Not directly, and possibly not at all. Googlebot ignores the Crawl-delay directive completely, so changing it has zero effect on how Google crawls your site. Any benefit you notice after removing it is really coming from faster discovery by Bing, Yandex, or other bots that do honor it, which can help indirectly if Bing traffic or reindexing speed matters to you.
Not with Crawl-delay. Google's documented method is to temporarily serve HTTP 500, 503, or 429 status codes when your server is under real stress; Googlebot interprets that as a signal to back off and retry later. Treat that as a short-term emergency measure while you address the underlying capacity problem, not a permanent configuration.
Google has moved away from offering manual crawl-rate controls, stating that its systems calculate an appropriate crawl rate automatically based on your server's health and response times. Use the Crawl Stats report in Search Console to monitor what Google is actually doing rather than looking for a dial to turn down.
There's no universally correct number, since bots interpret it differently, but if your server truly needs breathing room, start low, around 1 to 5 seconds, rather than the much higher values, 10, 20, or more, sometimes copied from old tutorials. A double-digit delay on a site with more than a few hundred pages can stretch a full recrawl by Bing or Yandex out to multiple days.
Does your site have this issue?
Run a free, AI-powered audit and we’ll flag this and 150+ other checks in about a minute. No signup.