Robots.txt

Robots.txt is a plain text file at the root of your site (yoursite.com/robots.txt) that tells search engine crawlers which parts of your site they are and are not allowed to crawl. Crawlers like Googlebot check this file before crawling.

It uses simple rules. For example, Disallow: /admin/ tells crawlers to stay out of your admin area, while Allow: permits access. You can also point crawlers to your XML sitemap from here. It is commonly used to keep crawlers away from areas that offer no value in search — login pages, internal search results, or duplicate parameter URLs.

Here is the misunderstanding that causes real damage: robots.txt controls crawling, not indexing. Blocking a page in robots.txt tells Google not to crawl it — but if that page is linked from elsewhere, Google can still index the URL (often with no description, since it never read the content). If you truly want a page kept out of search results, use a noindex tag on the page itself, not a robots.txt block.

Even more important is the catastrophic mistake it can cause. A single line like this:

• User-agent: *
• Disallow: /

tells every crawler to stay off your entire site. This line is standard on staging sites during development — and every so often it gets pushed live by accident, quietly wiping a business off Google. If your rankings suddenly vanish, checking robots.txt is one of the first things to do.

Because the stakes are high and the syntax is unforgiving, robots.txt is worth handling carefully. It is a core part of technical SEO basics, and it works hand in hand with indexing controls.