Why it matters
- Bot protection services, such as Cloudflare and AWS WAF, can answer a request with a challenge page instead of the page asked for. Cloudflare's documentation says a visitor cannot reach the page without passing its challenge, which runs JavaScript in the visitor's browser and may ask them to check a box or press a button; AWS WAF's challenge and CAPTCHA are scripts too.
- The site answered our bot with one, so a crawler that cannot pass the challenge gets it in place of your content. Whether an AI search crawler is challenged too depends on the rules that decide: they may let it through, or refuse it as they refused us.
- Protecting a site from bots is often the right choice, so this is information, not a fault: check that the crawlers you want reach your pages.
How to fix
- Decide which bots should reach your pages, such as search engines' and AI search services' crawlers, and let them through the rules that challenge.
- In Cloudflare, a custom rule placed first, with the expression
(cf.client.bot)and the Skip action, lets verified bots, such as search engine crawlers, bypass your other custom rules. Bot Fight Mode, on the Free plan, cannot be bypassed this way: turn it off, or upgrade to Super Bot Fight Mode (Pro and above). - In AWS WAF, the challenge comes from a rule whose action is CAPTCHA or Challenge: narrow what that rule matches.
- Arablyzer never tries to get past a challenge, so the checks of this page's content did not run.
How we detect
- We fetch the page as
ArablyzerBotand read its answer's headers, whatever its status. - A challenge is told apart by the signals its service documents alone: Cloudflare's
cf-mitigated: challengeheader, which it sets on every kind of challenge page, and AWS WAF'sx-amzn-waf-actionheader,challengewith the status202orcaptchawith405. - We guess nothing from the page's text, a plain refusal such as
403without these headers is not a challenge to us, and other services' challenges go undetected. - When the answer is a challenge, no other check takes it for the page: the report says the site answered with a challenge, and the page is neither rendered in a browser nor measured, since running the challenge's script could get past it. When our own request reaches the page but a browser is answered with a challenge, that browser's render stops at the answer and nothing more is asked of the site for it, and Lighthouse does not run.
- It is information, so the rule itself deducts nothing. A page the site answers with a challenge was not checked, though: the scan has no score, overall or by category, and is partial.