Crawl-delay in robots.txt: who obeys it

Crawl-delay asks a crawler to wait between requests, but Google doesn’t support it: its robots.txt spec names it as an unsupported field. The line isn’t part of RFC 9309 either, so each crawler decides for itself. These are the documented answers:

crawler     reads Crawl-delay        source, as dated on the page
Googlebot   no                       Google's robots.txt spec, 2026-08-31
Bingbot     yes                      Bing webmaster blog, 2012-05-03
ClaudeBot   yes                      Anthropic help center, 2026-04-07
CCBot       yes                      Common Crawl FAQ, undated
Applebot    no                       Apple support, 2026-09-04
Yandex      no, since 2018-02-22     Yandex Webmaster help, undated

User-agent: *
Crawl-delay: 10
Disallow: /search

The tester reads it as RFC 9309 would: Crawl-delay decides nothing, so for Bingbot /products/duck is allowed and /search?q=duck is disallowed by line 3. The delay shows next to the group, with a lint note that Google ignores it.

What the number means

Bing’s post defines it as a window of 1 to 30 seconds in which BingBot fetches at most one page; a delay of 5 caps it near 17,280 pages a day, and 10 at 8,640. Common Crawl reads Crawl-delay: 2 as at most one request every 2 seconds, and Anthropic’s own example is Crawl-delay: 1 under User-agent: ClaudeBot. A higher number means fewer pages crawled, so set it only as high as the server needs.

Gotchas

• Bing reads it from * as well. BingBot follows only its most specific group, but its post makes Crawl-delay the exception: a delay in the * group counts even when a Bingbot group exists. The tester only shows the delay of the group it applies.
• It doesn’t end a group. RFC 9309 (section 2.2.4) says records it doesn’t define mustn’t interfere with parsing, so a User-agent line after a Crawl-delay still joins the same group.
• Each host counts on its own. Bing manages the delay separately for every subdomain with its own robots.txt.
• Yandex and Bing have settings instead. Yandex points to the crawl rate setting in Yandex Webmaster. In Bing, a robots.txt delay overrides the crawl control in Bing Webmaster Tools.

Slowing Googlebot

Google’s crawl-rate guide (2026-10-06) says to answer with 500, 503 or 429 for a short time, a few hours up to a day or two. Google then slows down across the whole hostname and speeds up again when the errors stop. Kept up for days, the errors can drop URLs from the index. A 503 or 429 can say when to come back:

HTTP/1.1 503 Service Unavailable
Retry-After: 120

If serving errors isn’t possible, Google takes a special request about an unusually high crawl rate.