Robots.txt Tester

Check a robots.txt: would this crawler be allowed to fetch this URL, and which rule decided?

Allowed Googlebot is allowed to fetch /private/public/page.html — line 3: Allow: /private/public/.

The group for * applied — 2 rules in the merged group.

The file, line by line · 9 lines
  1. 1User-agent: *
  2. 2Disallow: /private/
  3. 3Allow: /private/public/
  4. 4
  5. 5User-agent: GPTBot
  6. 6Disallow: /
  7. 7
  8. 8Sitemap: https://quackyard.com/sitemap.xml
  9. 9
Lint · 1 finding
  • Line 8: Sitemap listed: https://quackyard.com/sitemap.xml

Judged per RFC 9309, the Robots Exclusion Protocol: the longest matching rule wins, Allow wins a tie, and * with $ wildcards are matched by hand. Everything runs in this page, nothing is uploaded.

More in the yard