How to Write a Robots.txt File That Actually Helps Your SEO

The robots.txt file is small, simple, and easy to get catastrophically wrong. A single misplaced rule can accidentally block search engines from crawling your entire site — and because nobody visits robots.txt manually, this kind of mistake can go unnoticed for weeks while your rankings quietly disappear. Here's how to write one correctly.

What Robots.txt Actually Does (and Doesn't Do)

Robots.txt is a plain text file at the root of your domain (yourdomain.com/robots.txt) that gives instructions to well-behaved web crawlers about which parts of your site they're allowed to crawl. Critically, it's a request, not an enforcement mechanism — legitimate search engines (Google, Bing) respect it, but it doesn't prevent access, it just asks crawlers not to visit certain paths. It's also not a security tool — anything you don't want publicly accessible should be protected with actual authentication, not just excluded from robots.txt, since the file itself is publicly visible and effectively tells anyone looking exactly which paths you consider sensitive.

Basic Syntax

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

User-agent specifies which crawler the rule applies to (* means all crawlers). Disallow specifies paths that shouldn't be crawled. Allow can be used to create exceptions within a disallowed directory. Including your Sitemap location helps crawlers discover your indexable pages more efficiently.

The Most Dangerous Mistake: Accidentally Blocking Everything

This single line, left in place from a staging environment, blocks your entire site from being crawled:

User-agent: *
Disallow: /

This happens more often than you'd expect — a rule added during development to keep search engines away from a staging site gets carried over into production during deployment and never removed. If your site's traffic has mysteriously dropped and you haven't changed anything else, check your robots.txt file first.

What You Should Actually Block

Reasonable candidates for disallowing:

• Admin or backend login pages (/wp-admin/, /admin/)
• Internal search result pages (which often create low-value, near-duplicate content at scale)
• Staging or testing subdirectories
• Duplicate content generated by filtering/sorting parameters, if not otherwise handled via canonical tags
• Shopping cart, checkout, and account pages that provide no SEO value and could otherwise be crawled unnecessarily

What You Should NOT Block

• Your CSS and JavaScript files — Google needs to render these to properly understand your page layout and mobile-friendliness. Blocking them can hurt rankings, since Google can no longer see your page the way a real visitor does.
• Any content you actually want to rank in search results (this sounds obvious, but accidentally blocking blog or product pages happens more than you'd think, especially with overly broad wildcard rules).

Testing Your Robots.txt

Before deploying changes, use Google Search Console's robots.txt testing tool to verify your rules behave as expected. This catches syntax errors or unintended blocking before it affects your live site's crawlability — testing after the fact means you're finding out about a mistake only once damage is already done.

Robots.txt vs Noindex: Know the Difference

These are often confused but serve different purposes. Disallow in robots.txt prevents crawling (the crawler won't even visit the page), while a noindex meta tag or header allows crawling but tells search engines not to include the page in search results. If a page is disallowed via robots.txt, search engines generally can't see a noindex tag on it either, since they never crawl it in the first place — meaning a disallowed-but-linked-to page can sometimes still appear in search results with no description, just because other sites link to it and Google knows it exists without being able to see the content.

For pages you genuinely want completely out of search results, noindex (while still allowing crawling) is often the more reliable approach than robots.txt disallow alone.

The Bottom Line

Robots.txt is powerful precisely because it's simple — which also makes mistakes easy and consequences serious. Review it after every major deployment, especially staging-to-production migrations, and test changes before they go live rather than discovering a blocking mistake weeks later through a mysterious traffic drop.