robots.txt wildcards: * and $

In a robots.txt rule, * matches any run of characters, including none, and a $ at the end means the URL has to end there. RFC 9309 (section 2.2.3) requires crawlers to support both, and Google’s spec says Google, Bing and other major search engines do. Everything else is matched literally, from the start of the path, query string included.

User-agent: *
Disallow: /*.pdf$
Disallow: /*?
Allow: /*?page=

What it matches

/files/report.pdf          Disallowed   line 2   ends in .pdf
/files/report.pdf?dl=1     Disallowed   line 3   the URL ends in ?dl=1, so line 2 misses
/files/report.PDF          Allowed               matching is case-sensitive
/shop?sort=price           Disallowed   line 3   any query string
/shop?page=2               Allowed      line 4   the longer rule wins
/shop?page=2&sort=price    Allowed      line 4
/shop                      Allowed

Which rule wins

When several rules match, the longest one decides, counted in octets, and Allow wins a tie (RFC 9309, section 2.2.2). Wildcards count as characters: Allow: /*?page= is 8 long and Disallow: /*? is 3, so pagination stays crawlable while other query strings don’t. The order of the lines doesn’t matter.

That’s also how a wildcard rule can lose to a plain prefix. In Google’s example, Allow: /page and Disallow: /*.ph are both 5 octets, so for /page.php5 the tie goes to Allow.

Patterns worth knowing

Disallow: /*.pdf$          URLs that end in .pdf
Disallow: /*?              any URL with a query string
Disallow: /*sessionid=     a parameter anywhere in the URL
Disallow: /*/private/      a private folder below the top level
Allow: /$                  the home page only, next to Disallow: /

Gotchas

• A trailing * adds nothing to the match. /fish* matches the same URLs as /fish, because rules already match prefixes. It does add to the length, so Allow: /fish* beats Disallow: /fish.
• $ only anchors at the end. In the middle of a rule it’s an ordinary $.
• There are no regular expressions. ., ?, + and [ ] are literal characters: /*.pdf$ means a dot, then pdf.
• Start every rule with / or *. Paths begin with /, so Disallow: private never matches anything.
• Case matters. Add Disallow: /*.PDF$ if your files use both spellings.