Definition
- A text file named
robots.txtin the top-level directory of a site, such ashttps://example.com/robots.txt, following the Robots Exclusion Protocol, which became a standard as RFC 9309 in 2022. - It is made of groups: one or more
User-agentlines naming crawlers, thenDisallowandAllowrules for paths. A crawler follows the group with its name, or else the*group, and the longest matching rule wins. - It applies only to the host, protocol and port it is served from:
https://example.com/robots.txtdoes not coverhttps://shop.example.com/orhttp://example.com/.
Why it matters
- It is used mainly to keep crawlers from overloading the site, not to hide pages: Google may index a blocked URL if other pages link to it, and show it in results without a description.
- Its rules are not protection: respectable crawlers follow them, others may not, and every path you list becomes known to anyone who reads the file.
- On Arabic sites, save the file as UTF-8, or Google may ignore characters outside it and the rule breaks. Each subdomain, such as
ar.example.comanden.example.com, needs its own file.
Example
A store's file that blocks the cart and internal search results, and points to the sitemap:
text
# https://example.com/robots.txt
User-agent: *
Disallow: /cart/
Disallow: /بحث/
Sitemap: https://example.com/sitemap.xmlGoogle compares rules with URLs in their percent-encoded form, so it treats Disallow: /بحث/ the same as Disallow: /%D8%A8%D8%AD%D8%AB/.
Common mistakes
Disallow: /in the*group, left over from the development site, blocking the whole site.- Blocking a page that has
noindexin robots.txt: Google never sees the rule, and the URL may stay in results. - Writing
noindexorcrawl-delayin the file: Google supports neither. - Answering the request for the file with a
5xxerror: Google stops crawling the whole site for the first 12 hours.