Robots.txt and XML Sitemaps: The Complete Technical SEO Guide

What Robots.txt Actually Controls (and What It Doesn't)

Robots.txt is a plain text file at the root of your domain that tells well-behaved crawlers which parts of your site they are allowed to request. It is organized into user-agent groups, each with a set of Allow and Disallow rules, and it can optionally point crawlers to your sitemap's location.

What it does not do is remove pages from search results, and it does not enforce anything — it's a request, not a lock. A disallowed page that already has external links pointing to it can still appear in search results, just without a description, because the crawler respects the restriction but may already know the page exists from other signals.

Common Robots.txt Mistakes That Deindex Entire Sites

The most damaging mistake is a leftover "Disallow: /" from a staging environment that makes it into production unnoticed, blocking the entire site from every crawler. This happens more often than it should, usually after a site migration or redeploy where the staging robots.txt gets copied over by mistake.

Subtler mistakes include accidentally disallowing a path that contains pages you actually want indexed — blocking /blog/ when you meant to block only /blog/drafts/ — or disallowing a CSS or JavaScript directory that search engines need to fetch in order to render your page correctly.

• Check for a stray "Disallow: /" left over from staging
• Make sure disallow rules don't accidentally cover content you want indexed
• Don't block CSS/JS paths search engines need to render your pages
• List your sitemap location with the Sitemap: directive

What an XML Sitemap Does for Crawl Priority

A sitemap is a structured list of your site's URLs, optionally annotated with when each page was last modified, how often it changes, and its relative priority. It doesn't force a search engine to index everything listed, but it gives crawlers a direct map instead of relying entirely on discovering pages through internal links — which matters most for new pages, pages buried deep in your site structure, or pages with weak internal linking.

For a large site, a sitemap is often the difference between a new page getting crawled within hours versus weeks. Search engines use it to prioritize what to crawl next, especially on sites they don't already crawl frequently.

Sitemap Mistakes: Stale Dates, Broken URLs, and Missing Pages

A sitemap with lastmod dates that update on every single build — even for pages whose content hasn't actually changed — trains search engines to stop trusting that field, since it stops correlating with real freshness. It's better to only update lastmod when content meaningfully changes.

Sitemaps that include 404s, redirected URLs, or pages blocked by robots.txt send mixed signals and waste crawl budget on URLs that will never get indexed. And on the flip side, a sitemap missing entire sections of a growing site — new tools, new categories, new posts — means those pages are relying entirely on internal links to ever get discovered.

Validating Your Robots.txt

Before assuming your robots.txt is correct, check what it's actually saying. A single misplaced rule can sit unnoticed for months while quietly blocking crawlers from content you want indexed.

Validating Your Sitemap

The same applies to your sitemap — it should be checked for valid XML, a reasonable URL count, and sane lastmod dates, not assumed correct just because it was generated automatically.