A robots.txt file is a public note at the root of a host. Google's introduction is blunt about what it is not: it is not a way to keep a web page out of Google. It manages crawler traffic. To keep a page out of Search, use noindex or protect the page with a password. Most of the indexing accidents on publishing sites come from ignoring that sentence.

The file is also easy to ship by mistake. A staging rule, a pattern that is one character too short, or a disallow on the CSS and JavaScript the article needs will change what Google can fetch. The page can remain in the index anyway, often without a snippet, which is the outcome people were trying to avoid.

What Google actually reads

The file must be named robots.txt, saved as UTF-8 text, and placed at the root of the host, protocol, and port it applies to. https://example.com/robots.txt does not govern https://www.example.com/, http://example.com/, or https://m.example.com/. Each of those hosts needs its own file if you need rules there. A file in a subdirectory is not a robots.txt file. Crawlers do not look for one there.

Google enforces a size limit of 500 kibibytes. Everything after that mark is ignored. If a generated file has grown past that because every parameter combination has its own line, consolidate the rules into directory patterns instead of adding more lines.

Field names are case-insensitive. Path values are case-sensitive. Disallow: /file.asp does not match /FILE.asp. A # starts a comment. Google's crawlers support user-agent, allow, disallow, and sitemap. Other fields, including crawl-delay, are not supported. Google ignores invalid lines and will try to parse rules out of a file even when the rest of the body is unexpected markup.

A user-agent matches one group: the most specific group that names that crawler. A Googlebot group is not merged with the * group. If you write two groups for the same user-agent, Google combines those specific groups with each other. The * token matches crawlers except the AdsBot crawlers, which have to be named on their own if you need to restrict them.

For a path, the most specific matching rule wins, judged by the length of the rule path. When two rules conflict, Google uses the least restrictive one. Disallow: /fish matches any path that starts with /fish, including /fishheads and /fish.html. Disallow: /fish/ matches the folder and does not match /fish.html. A trailing * does not make a prefix more specific. $ anchors the end of the URL. These are the rules in Google's robots.txt specification, and they are why a short pattern hides more than the author intended.

Google caches robots.txt, generally for up to 24 hours, and may keep a cached copy longer when it cannot refresh the file. A 5xx on robots.txt is serious: for the first 12 hours Google stops crawling and keeps retrying; then it may use the last good file for up to 30 days while still retrying. A 4xx other than 429 is treated as "no robots.txt", which means no crawl restrictions. Do not use 401 or 403 on robots.txt to slow crawling. It does not limit the crawl rate, and it can remove your rules entirely.

Mistake: blocking a page to remove it from Search

If Google cannot crawl a URL, it cannot see a noindex on that URL. Links from other pages can still put the URL into Search, often with no description. Search Console calls this "Indexed, though blocked by robots.txt".

To drop a page from the index, allow crawling and add a noindex meta tag or an X-Robots-Tag header. After the URL has left the index, you can decide whether a disallow is still useful to save fetches. Using disallow as the removal method leaves the URL in the report and in Search.

noindex written inside robots.txt is not a supported field. Google stopped honoring an unsupported noindex directive in robots.txt years ago. The line is ignored with the other unsupported fields. Put noindex on the page.

Mistake: a staging disallow on the production host

Staging sites often ship Disallow: / so they stay out of Search. When that file is copied to production, Google fetches robots.txt, sees a total disallow, and stops crawling the host. Because of the cache, the recovery is not instantaneous even after you upload a corrected file.

Make the production file a deploy artifact you can read. A useful production file blocks the paths that are genuinely low value, allows everything you want rendered, and lists the sitemap:

User-agent: *
Disallow: /search
Disallow: /account/
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Add a check that fails the deploy if the production file disallows / for Googlebot or for *. Open the live file in a private window after release. Search Console has a robots.txt report for a file that is already publicly readable. Developers can also test files with Google's open-source robots.txt parser. There is nothing to "submit" for a corrected file beyond fetching it; Google recrawls robots.txt on its own. If you need the cached copy refreshed, use Search Console's process for an updated robots.txt rather than waiting out the full cache window.

Mistake: blocking the files that build the page

Google renders pages. If robots.txt disallows the scripts or styles the article needs, the rendered page can look empty even though a browser with a full cache looks fine. Google's intro says not to block resources when the page becomes hard to understand without them. Media files are different: a disallow can keep an image or video out of those search surfaces, and other pages can still link to the file.

After any robots.txt change, run a live test on an article in URL Inspection and look at the screenshot and the list of resources that failed. The URL Inspection guide shows where that evidence is. If the screenshot is blank, unblock the resources and test again before you touch the article template.

Mistake: a pattern that is broader than the directory

Disallow: /p blocks /pricing, /press, and /products, not a folder named p. Disallow: /*? blocks every URL with a query string, including tracking links you did not mean to hide and any filtered URL you did mean to index. Disallow: /*.pdf$ blocks PDFs only when the path ends in .pdf. Drop the $ and you also match some paths that merely contain .pdf.

Write the folder with a trailing slash. Test the pattern against a list of real paths from your sitemap: the ones that must be crawled, and the ones that must not. The sitemap guide is the list of URLs you claim are indexable. A URL that is in the sitemap and disallowed in robots.txt is a contradiction. Google may still index the URL from links and will not see a canonical or a noindex on it.

Mistake: using robots.txt as a canonical or a soft-404 fix

Disallowing a duplicate does not consolidate it. Google's canonical documentation says not to use robots.txt for canonicalization, because the URL can be indexed without its content. Let Google fetch the duplicate, see the canonical, and combine the URLs. The canonical guide is the path that actually names a preferred URL.

Disallowing a soft 404 keeps the bad URL in the crawl queue longer than a real 404 would. The crawl budget guide says a 404 is a strong signal to stop fetching that URL, while a blocked URL stays queued and is fetched again when the block is lifted. Return the status code. The soft 404 guide covers the patterns.

Do not use robots.txt to "lend" crawl budget to other pages for a week. Google's crawl budget guide says not to use it as a temporary reallocation tool. A disallow is for URLs you do not want crawled at all. Freed capacity is not automatically spent on the rest of the site unless Google is already at the host's crawl capacity limit.

Mistake: assuming one file covers the brand

Each host has its own crawl rules and, in Google's crawl model, its own crawl budget. A redesign that moves articles to www while robots.txt lives only on the apex host leaves the www host unrestricted or, worse, leaves an old www file in place. Check every host that still answers: apex, www, and any content subdomain. Confirm the sitemap line uses the same host you canonicalise to.

robots.txt is public and advisory. Anyone can read it. It is not access control. Private drafts and subscriber-only files need authentication. A disallow only asks polite crawlers to skip the path, and other clients may ignore the file entirely.

After the file changes

Fetch /robots.txt on the production host. Inspect one URL you want indexed and one URL you want skipped. The first should show crawling allowed and a rendered article. The second should show crawling blocked, and you should be sure you do not need noindex on it instead. Watch the Page indexing report for a rise in "Blocked by robots.txt" on paths you meant to keep, and for "Indexed, though blocked by robots.txt" on paths you meant to remove.

The file is a crawl boundary. Pages you want in Search have to sit on the allowed side of it, with noindex used only where you truly want them out. If a section vanished after a launch, read the live robots.txt before you rewrite the articles.

CrawlFaster's status tracking is a way to notice a sudden drop in crawled URLs after a configuration change. The robots.txt file on the host is still the document that caused it.