Disallow a folder in robots.txt

Disallow: /private/ under a User-agent line blocks the /private/ folder and everything below it. Rules match from the first character of the URL path (RFC 9309, section 2.2.2), so the rule covers every path that starts with /private/.

User-agent: *
Disallow: /private/

What it matches

/private/report.pdf    Disallowed   line 2
/private/              Disallowed   line 2
/private               Allowed      the folder's URL without its slash
/private-notes/        Allowed
/Private/report.pdf    Allowed      paths are case-sensitive
/docs/private/x        Allowed      rules match from the start of the path

Without the slash, Disallow: /private also blocks /private-notes/ and /privateer, since both start with /private. For folders of that name below the top level, add Disallow: /*/private/; it doesn’t cover /private/ itself, so keep both lines.

Exceptions

A longer Allow beats a shorter Disallow, whatever their order in the file:

User-agent: *
Disallow: /private/
Allow: /private/press/

/private/press/kit.zip is allowed by line 3, and /private/report.pdf is still disallowed by line 2.

What it doesn’t do

• It doesn’t protect the folder. RFC 9309 (section 3) warns that the file is public and so advertises the very paths it lists. Put private files behind authentication.
• It doesn’t remove pages from search. Google can still index a disallowed URL it finds through links, and show it without a snippet. To drop a page, let it be crawled and send noindex, as a <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex header. Google says it can’t see either on a page that robots.txt blocks, and that it doesn’t support noindex inside robots.txt.

For files without HTML, like PDFs, the header is the only way. Send it for the folder and leave the folder crawlable:

location /private/ {
    add_header X-Robots-Tag "noindex" always;
}

<Location "/private/">
    Header set X-Robots-Tag "noindex"
</Location>