Why it matters
- A visitor who follows a broken link reaches an error page, without what the link promised, and may leave.
- Google relies on links to judge how pages relate and to find new pages to crawl. A link that leads to an error leads nowhere.
- Google does not index addresses that answer a
4xxstatus code. A5xxerror asks Google's crawlers to slow their crawling for a while.
Example
<a href="/ar/ofers/">العروض</a>
<a href="/ar/offers/">العروض</a>
How to fix
Correct the address in the link, or restore the page it leads to. If the page has moved to a new address, send its old address there with a permanent redirect, so every old link to it works:
nginx
location = /ar/ofers/ {
return 301 /ar/offers/;
}- Remove links to pages that are gone for good.
- If the error is a
5xx, the fault is in the server: look for it in its error logs. - In Apache (mod_alias):
Redirect permanent /ar/ofers/ /ar/offers/.
How we detect
- We read the
<a href>and<area href>links in the page's HTML as the server sends it, before JavaScript runs, completed against the page's base address. We check those that lead to the page's own origin, that is the same scheme, name and port: links to other sites, or to another name of the same domain such aswww, are not checked. Each address is checked once, without what follows#; the page itself, and addresses with a user name or password, are left out. - We never ask for a link in a path that robots.txt disallows for
ArablyzerBot: by a group that names it or, where none does, by theUser-agent: *group. The page itself is a visit someone asked for, so only a group namingArablyzerBotstops it; its links are requests of the scan's own, made as a crawler would, so they follow the*group too. - We count 1,000 links at most on a page, and check the first 50 links, four at a time, within 20 seconds in all, and each request within 10 seconds (a link can take two requests, a
HEADand aGET). The report says how many links were not checked, and “at least” when the page has more than 1,000. - We ask for each link with
HEAD; when it answers a4xxor5xxerror, or the connection fails, we ask withGET, as a visitor's browser does, and theGETstatus decides. A timeout, or an address the egress proxy refuses, leads to noGET. We follow no redirect, so a link that answers a3xxredirect works for us, and we read no response's content. Each request goes through the egress proxy that vets every address, as the page's own does. - The rule fails for each link that answers a status from
400to599, except401,403,407,429and503. Those five are how a site turns away a visitor it takes for a bot, one that must sign in, or one it cannot serve just now, and they say nothing of whether the link works, so we do not judge them. Nor do we judge a timeout or a failed connection. After the first429, with which a server asks for fewer requests, we ask for no more links. The report says how many links got no answer. When no link got one, the rule could not run. - We check the scanned page's links alone, and crawl none of the site's other pages.