An XML sitemap is a list of the URLs you would like search engines to know about. Google's sitemap documentation is explicit that submitting one is a hint: it does not guarantee that Google will download the file or crawl the URLs in it. A clean sitemap still earns its place. It is the list you control, it can carry a lastmod date Google will use when the date is trustworthy, and it is how the Page indexing report can be filtered down to the URLs you meant to publish.

A messy sitemap does the reverse. It asks Google to fetch redirects, error pages, and copies you have already canonicalised somewhere else, and it makes the Search Console charts harder to read because the "submitted" set is not the set you care about.

Limits, encoding, and where the file can live

Google supports the formats in the sitemaps protocol. For a single sitemap file, every format is limited to 50,000 URLs and 50MB uncompressed. Past either limit, split the list and point a sitemap index file at the pieces. You can submit more than one sitemap or index. That is useful when you want Search Console to show each file separately.

The file must be UTF-8. URLs inside it must be absolute. Google crawls the URL as written, so /news/story is wrong and https://www.example.com/news/story is right. Tag values in XML have to be entity-escaped.

You can host the file in a subdirectory. Unless you submit it through Search Console, a sitemap only covers descendants of the directory it sits in. A sitemap at the root can cover the host, which is why the root is the usual place. A sitemap on one host does not cover a subdomain. blog.example.com needs its own file or an entry that you are allowed to submit for that property.

Google also accepts RSS 2.0 and Atom 1.0 feeds as sitemaps, and an mRSS feed can describe video. A feed is a weak fit as your only sitemap because it usually lists recent URLs, not the whole site. A plain text sitemap is allowed too: one absolute URL per line, nothing else, and a .txt name. XML is the format that can also carry image, video, news, and hreflang annotations when you need them.

Only the URLs you want as canonicals

Google treats URLs in a sitemap as suggested canonicals. The signal is weaker than a redirect or a rel="canonical" link, and it still conflicts with those stronger signals when the lists disagree. Include a URL only when all of the following are true:

  • It is the preferred URL, not a tracking-parameter copy, not a print view, not the HTTP version.
  • It returns 200. Redirects, 404s, and soft 404s do not belong in the list.
  • It is indexable: no noindex, and not disallowed in robots.txt.
  • It is the page you want people to land on from Search.

If the same article exists at two URLs, list one. If mobile and desktop are separate URLs and you only want one in Search, list that one. Listing both without the correct annotations tells Google to consider both as candidates.

Drop URLs the day they are deleted or merged. A sitemap that still advertises a retired campaign URL will keep sending fetches to a soft 404 or to a redirect. Generate the file from the CMS or the database so publication and removal change the list. A hand-edited file drifts the first week someone forgets it.

lastmod is the only date field worth getting right

Google ignores priority and changefreq. Leaving them in the file is harmless. Tuning them does nothing for Google.

Google uses lastmod when it is consistently accurate and can be checked, for example against the actual modification of the page. The value should be the date and time of the last significant update: the main content, the structured data, or the links on that page. Updating the copyright year in the footer is the example Google gives of a change that is not significant.

Set lastmod from the article's real edit time. Do not stamp every URL with today's date on every deploy. A site that does that has taught the field to mean "the sitemap was rebuilt", and Google's guide says the value is used when it is verifiably accurate. A meaningless stamp fails that test.

Do not update lastmod on an article because a sidebar widget changed, unless that widget is a meaningful part of the page. Do update it when you correct the story, add a section, or change the canonical target.

Split by type so the report is readable

A sitemap index that groups URLs by template makes problems obvious:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://www.example.com/sitemaps/articles.xml</loc></sitemap>
  <sitemap><loc>https://www.example.com/sitemaps/sections.xml</loc></sitemap>
</sitemapindex>

Submit the index in Search Console. The Sitemaps report shows when the file was last read and whether it parsed. The Page indexing report can be filtered by sitemap, so you can see articles and section pages separately. A spike in "Crawled – currently not indexed" that exists only in the tag sitemap is a tag-template problem. The same spike in the article sitemap is an article problem. The status names are explained in the Page indexing guide.

Keep each child file under the size limits. The order of URLs inside a file does not matter to Google.

Submit it in more than one way

  • Submit the index in Search Console under Sitemaps. You can also submit with the Search Console API.
  • Add Sitemap: https://www.example.com/sitemap.xml in robots.txt. The URL must be absolute. Google does not assume a www or https variant. You can list more than one sitemap line. There is no documented limit on how many sitemap lines robots.txt may contain. The sitemap field is not tied to a user-agent group.
  • If you use Atom or RSS, WebSub can broadcast changes to search engines that support it, including Google. That is a feed feature, not a replacement for a full sitemap.

Cross-site sitemaps are allowed when you have verified the properties. One file on a host you control can list URLs from other verified sites, or you can host per-site files in one place and point each site's robots.txt at its own file. Do not list URLs for a host you have not verified and then wonder why Search Console rejects the file.

News, images, and languages

A news sitemap is a separate convention for news publishers, limited to recent articles. It does not list the evergreen archive. Keep the normal sitemap for the rest of the site. Putting a three-year-old feature in the news sitemap does not make it news.

Image and video extensions belong on the page URL that shows the media, not on a naked file URL you do not want as a landing page. hreflang annotations in a sitemap are an alternative to hreflang in HTML. If you use both, they should name the same alternates. A canonical in another language than the hreflang cluster is a conflict. Language canonicals are part of the canonical guide.

Build the file so the next deploy cannot lie

Generate on publish. When an article is updated, change its lastmod and leave the others alone. When it is unpublished, remove the URL. Exclude noindex URLs with a query, not with a hope that nobody will notice. Exclude parameter URLs in the generator so a tracking template cannot append them later.

After deploy, open the sitemap in a browser. It should be XML, not an HTML error page, not a login page, and not a 500. Then look at Search Console's Sitemaps report for parse errors. A file that returns 200 and a styled website is a file Google cannot parse.

Pair the sitemap with links. Pages that are only in the sitemap and never linked are easier to leave unfetched. The internal linking guide is the other half of discovery. Pages that are linked and also listed are the pages you can defend.

A review you can run monthly

Open each sitemap in Search Console and note submitted URLs versus indexed URLs. For the file that looks worst, filter the Page indexing report to that sitemap and sort the reasons. Fix the top reason in the template that generates those URLs. Do not resubmit the sitemap ten times in a day. A resubmit after a real correction is enough to ask Google to read the file again.

The sitemap is the inventory you claim is worth crawling. If that inventory includes redirects and empty tags, the crawl will include them too. Clean the generator, keep lastmod tied to real edits, and use the per-sitemap filter the next time a section quietly stops being indexed.

CrawlFaster can import sitemap URLs and follow their indexing status, which is a practical way to see one child sitemap fall behind the others. The file those URLs come from still has to be generated from the pages you mean to keep.