Why it matters
- Search engines read a sitemap to find a site's pages. When Google cannot fetch a sitemap or parse it, it tries again for a few days and then stops; when it can fetch one with errors, it queues for crawling only the URLs it could parse without errors.
- A Sitemap line of robots.txt must give a full URL, including its protocol and host, as Google's robots.txt specification says: a path alone, such as
/sitemap.xml, is not one. - Google reads sitemaps in the protocol's XML, as RSS 2.0 or Atom 1.0 feeds, and as text with one URL on each line. An XML sitemap's root element is
urlset, and a sitemap index'ssitemapindex, both in the namespacehttp://www.sitemaps.org/schemas/sitemap/0.9, written exactly: Search Console reports another as an incorrect namespace. - As in all XML, characters such as
&in a URL must be escaped. Google says an unescaped character in a URL is often what stops it from parsing a sitemap. - A sitemap may be compressed with gzip, as a file such as
sitemap.xml.gz, and must then decompress as a whole. Search Console reports one that does not as a compression error.
Example
User-agent: * Disallow: /cart/ Sitemap: /sitemap.xml
User-agent: * Disallow: /cart/ Sitemap: https://shop.example/sitemap.xml
How to fix
- Give each Sitemap line of robots.txt the sitemap's full URL:
text
Sitemap: https://www.example.com/sitemap.xml- Open that address yourself: it should answer with the sitemap, not an error or an HTML page. If it does not, fix the address in robots.txt, or put the sitemap there.
- Start an XML sitemap with the protocol's root element and namespace, and write each
&in a URL as&:
xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://www.example.com/ar/offers?page=2&sort=new</loc>
</url>
</urlset>- The other characters XML escapes are
'as',"as",>as>and<as<. - A text sitemap has one full URL on each line, and nothing else.
- Compress a sitemap as a whole, as
sitemap.xml.gz, and serve that file as it is. Ifgunzip sitemap.xml.gzdoes not give the sitemap back on your own machine, it will not for a search engine: write it again.
How we detect
- We read each Sitemap line of robots.txt. A value that is not a full
https://orhttp://URL fails at once. - We fetch the first 3 sitemaps robots.txt names as full URLs, or
/sitemap.xmlwhen it names none, asArablyzerBot, following redirects. We read up to25 MBof each, decompressing a file that is itself gzipped, and wait no more than ten seconds on the network for them all. We do not fetch the sitemaps an index lists, nor any that robots.txt names after those. - A sitemap robots.txt names fails when it answers with an error status, such as
404or410, or an HTML page. When robots.txt names none, an error or an HTML page at/sitemap.xmlmeans there is no sitemap there, which the rulesitemap-missingreports; a sitemap there is judged as the others are. - XML must be well-formed, with XML's own entities alone and one root element, and its root must be
urlsetorsitemapindexin the protocol's namespace, or an RSS 2.0 or Atom 1.0 feed. A byte order mark and white space before the XML are allowed, as Search Console allows them. Reading stops, and the file fails, at elements nested more than 32 levels deep, at an element with more than 256 attributes, or at a name, value or comment far longer than a sitemap needs: a sitemap has none of these. Anything that is not XML is read as a text sitemap: each line that is not blank must be a full URL. - A
urlset, asitemapindexor a text sitemap that lists nothing fails too. A sitemap past25 MBis judged in its first25 MB; the protocol allows50 MB. - A sitemap whose file is gzipped, as its first bytes say whatever its name, is decompressed. A file that starts as gzip and will not decompress fails: it is Search Console's compression error.
- We do not check the URLs a sitemap lists, nor its dates.
- When a sitemap cannot be read for a reason other than the site's answer, such as a failed connection, an address the scan does not reach, or a robots.txt that asks
ArablyzerBot, or every crawler withUser-agent: *, not to fetch it, the rule says it could not check, and does not fail. So it does when the site turns the scan away or cannot answer, with401,403,407,429or a server error (5xx), as RFC 9309 has a crawler treat a robots.txt that answers so: the site has not said the sitemap is missing.