Allow only Googlebot in robots.txt

To let only Google’s search crawler in, give Googlebot a group of its own that allows everything, and disallow everything under *:

User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /

A crawler obeys one group: the one that names it, or the * group when none does (RFC 9309, section 2.2.1). So Googlebot gets Allow: / on line 2 and never sees the Disallow: /, while Bingbot and DuckDuckBot are disallowed by line 5. An empty Disallow: in place of Allow: / does the same.

Who else this shuts out, or doesn’t

The * group catches every token without a group of its own, Google’s other tokens included:

• Google-Extended falls under *, so this file also opts the site out of Gemini training and grounding. Give it a group with Allow: / if you want those.
• Googlebot-Image and Googlebot-Video list Googlebot as a second token in Google’s crawler docs, and a crawler only has to match one of its tokens. They follow the Googlebot group here.
• Applebot follows Googlebot’s rules when no group names Applebot, Apple’s docs say, so it gets in too. Add User-agent: Applebot with Disallow: / to keep it out.
• Bing and the AI crawlers are disallowed.

The tester checks one product token at a time and applies RFC 9309 only, so it reports Googlebot-Image and Applebot as disallowed. The vendors’ own fallbacks above are what let them in.

Crawlers that never read *

Google documents that AdsBot-Google, AdsBot-Google-Mobile, Mediapartners-Google (AdSense and Ad Manager) and APIs-Google ignore the * group. The file above doesn’t stop them; if you don’t run Google Ads or AdSense and want them out, name them:

User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: Mediapartners-Google
Disallow: /

Gotchas

• Anyone can claim to be Googlebot. robots.txt asks; it doesn’t check. A server rule that trusts the User-Agent header lets in anything that sends Googlebot, so verify by reverse DNS or against the IP ranges Google publishes.
• The file itself stays readable. RFC 9309 makes /robots.txt implicitly allowed, so every crawler can still fetch the file that turns it away.
• Rules apply per host. shop.example.com needs its own copy.