Quackyard
  • Loading…

Robots.txt Tester

Check a robots.txt: would this crawler be allowed to fetch this URL, and which rule decided?

Allowed Googlebot is allowed to fetch /private/public/page.html — line 3: Allow: /private/public/.

The group for * applied — 2 rules in the merged group.

The file, line by line · 9 lines
  1. 1User-agent: *
  2. 2Disallow: /private/
  3. 3Allow: /private/public/
  4. 4
  5. 5User-agent: GPTBot
  6. 6Disallow: /
  7. 7
  8. 8Sitemap: https://quackyard.com/sitemap.xml
  9. 9
Lint · 1 finding
  • Line 8: Sitemap listed: https://quackyard.com/sitemap.xml

Judged per RFC 9309, the Robots Exclusion Protocol: the longest matching rule wins, Allow wins a tie, and * with $ wildcards are matched by hand. Everything runs in this page, nothing is uploaded.

Runs in your browser. Nothing you enter is sent anywhere.

How to use it

  1. Paste the file into robots.txt. It’s checked in your browser as you type.
  2. Put the page into URL to test: a full URL, or a path starting with /. The path and query are tested; the host and any #fragment are dropped.
  3. Pick a Crawler, or Another crawler… to type any other Product token, as it names itself, such as DuckDuckBot.

The verdict reads Allowed or Disallowed, with a sentence naming the deciding rule and its line. Below it are the group that applied, the file with the deciding line highlighted, and Lint: lines that are ignored or worth a second look.

Examples

The sample file, with URL to test /private/public/page.html:

User-agent: *
Disallow: /private/
Allow: /private/public/

User-agent: GPTBot
Disallow: /

For Googlebot the verdict is Googlebot is allowed to fetch /private/public/page.html — line 3: Allow: /private/public/. Both rules in the * group match, and the longer one wins. GPTBot has a group of its own, so the * group doesn’t apply to it at all, and it is disallowed by line 6, Disallow: /.

Wildcards, with Googlebot:

User-agent: *
Disallow: /*.pdf$

/files/report.pdf is disallowed, but /files/report.pdf?download=1 is allowed: the query is part of what’s matched, and $ pins the pattern to the very end.

Letting one crawler in and keeping the rest out:

User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /

Googlebot gets No rule in the group for Googlebot matches /page: Googlebot may fetch it. An empty Disallow: blocks nothing but still closes the User-agent lines above it, so the * group stays separate. DuckDuckBot is disallowed by line 5.

A rule written without its colon, Disallow /private/, is skipped. Lint shows Line 2: No colon after the field name, so the line is skipped: write it as Field: value., and /private/report.pdf stays allowed.

Good to know

  • The rules are RFC 9309, the Robots Exclusion Protocol. The longest matching rule decides, counted in octets. When an Allow and a Disallow are equally long, Allow wins. When no rule matches, or no group applies, the URL is allowed.
  • A crawler obeys one group. User-agent lines in a row share the rules below them, and groups naming the same crawler are merged. The * group is used only when no group names the crawler.
  • Crawler names ignore case and are matched whole. googlebot finds a Googlebot group, but Googlebot-Image is a different token. Enter the product token, not the full User-Agent header. Paths do match case-sensitively.
  • Rules match from the start of the path. Disallow: /admin also blocks /admin-login and /administrator; write /admin/ to block only the folder. * matches any run of characters, and $ at the end of a rule means the URL must end there.
  • Percent-encoding is compared, not guessed. %7E and ~ are the same character, so they match each other, but %2F is not a slash. Non-ASCII characters are compared as their UTF-8 escapes.
  • /robots.txt itself is always allowed, as RFC 9309 says, whatever the rules are.
  • Only User-agent, Allow and Disallow decide anything. Sitemap lines are listed under Lint, Crawl-delay is shown but Google ignores it, and other fields are flagged as unknown.
  • Disallowing a URL doesn’t hide it. Search engines can still list a URL other pages link to; use a noindex tag or a password for that.
  • The file can be up to 500 KB, the URL 2,048 characters and the product token 200. The page draws the first 400 lines but judges them all.

Questions

How do I test if a URL is blocked by robots.txt?

Paste the site’s robots.txt, enter the URL and pick the crawler: the verdict says allowed or disallowed and quotes the deciding line. Fetch the file with curl https://example.com/robots.txt. Every scheme, host and port has its own file, so blog.example.com doesn’t follow the one on example.com.

How do I block AI crawlers in robots.txt?

Give each one a group with Disallow: /: User-agent: GPTBot for OpenAI, ClaudeBot for Anthropic, CCBot for Common Crawl. Google-Extended keeps pages out of Google’s AI training without affecting Search. Robots.txt is a request, so it only stops crawlers that honor it.

Why does Disallow: / block everything?

Because every path starts with /, and a rule matches any path it is a prefix of. To block nothing, leave the value empty, Disallow:. To keep one folder crawlable, add a longer Allow rule such as Allow: /public/, which wins over the shorter Disallow.

Is robots.txt case sensitive?

Partly: field names such as Disallow and crawler names such as Googlebot aren’t, but paths are. Disallow: /Photos/ doesn’t block /photos/. The file itself must be named robots.txt in lowercase, at the top of the site.

Common answers