Robots.txt Guide — How to Control Googlebot and Other Crawlers

Updated October 2026 · 4 min read

The robots.txt file sits at the root of your domain (yourdomain.com/robots.txt) and tells crawlers which parts of your site to visit and which to ignore. Googlebot checks it before crawling anything on your site.

Robots.txt Syntax

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/

User-agent: Googlebot
Allow: /

Sitemap: https://example.com/sitemap.xml

• User-agent: * — applies to all crawlers
• Disallow: /path/ — blocks crawling of that path
• Allow: /path/ — overrides a broader Disallow for a specific path
• Sitemap: — tells crawlers where to find your XML sitemap

Common Mistakes That Hurt SEO

• Blocking CSS/JS files: If Googlebot cannot load your stylesheets, it cannot render your pages — which hurts Core Web Vitals scores. Never block /assets/ or /static/.
• Disallowing your whole site: Disallow: / blocks everything. Common accident when copying templates.
• Confusing robots.txt with noindex: robots.txt stops crawling, but does not remove a page from the index. A page already indexed stays indexed. Use <meta name="robots" content="noindex"> to remove indexed pages.
• Case sensitivity: Paths are case-sensitive. Disallow: /Admin/ does not block /admin/.

What robots.txt Cannot Do

Robots.txt is advisory — well-behaved bots follow it, but malicious scrapers do not. It also provides no privacy: the file is publicly readable, so listing a path in Disallow tells the world that path exists. For genuinely private content, use authentication or server-level access controls.