What a scan does
- Reads robots.txt first, and never asks for a page that a group naming ArablyzerBot disallows: the scan stops, and the report says so, with the rule and its line. Before it follows a redirect to another site, it reads that site's robots.txt too. A
User-agent: *group alone does not stop a scan, since a scan someone asks for is a visit, not a crawl, and a robots.txt that cannot be read does not stop it either. Every scan reads robots.txt afresh. - Fetches the page through a single egress proxy that refuses private addresses and the server's own, and follows up to ten redirects.
- Reads the page's HTML as it arrived, before any script: title, description, language, direction, links, structured data and forms.
- Asks DNS for the mail records when its rules need them: the TXT records of the page's organizational domain and of its
_dmarcname, and nothing else, within ten seconds at most. On the hosted service the two questions go as DNS over HTTPS (RFC 8484) to Cloudflare's resolver (https://cloudflare-dns.com/dns-query), or to the resolver the service is configured with, through the same egress proxy as the rest of its traffic: the domain's name goes to that resolver, and nothing else of the page. A scan that runs on a machine without an egress proxy, such as the command-line tool, asks that machine's DNS servers. When a scan has no way to ask DNS, it leaves those rules out, and the report says so. - Asks for the page's links to its own site when the broken-link rule needs them: it counts 1,000 links at most on a page, and asks for the first 50 links to the page's own origin, four at a time, within 20 seconds in all, and each request within 10 seconds (a link can take two requests), that robots.txt does not keep from ArablyzerBot, by a group that names it or by the
User-agent: *group, since the links are not the visit someone asked for, each with aHEADrequest, then aGETwhere it answers an error or the connection fails, through the same egress proxy, following no redirect and reading no content. - Renders the page in browsers when asked: Chromium and Firefox, and WebKit on servers with an isolated network. In each engine it measures Arabic text, direction, fonts and form fields, runs a curated set of axe-core's accessibility rules, and reads the stylesheets and fonts the browser loaded. No browser sends anything the page asks it to send: it refuses every request that is not
GETorHEAD, wherever it goes, and closes every WebSocket before it opens, so no form,sendBeaconorPOSTrequest goes out; it contacts at most 50 different hosts; and the report says what was refused. - Runs the rules: each is a fixed function that reads what was collected and fetches nothing, so the same page gives the same result every time.
Each rule's result
| Status | Meaning |
|---|---|
| Passed | The page has what the rule checks, and it found no problem |
| Failed | It found at least one problem; the report says where, with evidence |
| Needs review | What it found needs a person's eye, such as text over an image whose contrast cannot be measured; never deducted |
| Not applicable | The page has nothing the rule checks, such as the image rule on a page without images |
| Could not run | It did not finish, such as when rendering ran out of time, or the files the page loaded could not be read; left out of the score, which becomes partial |
The score
Each rule has a severity, and each severity a weight: critical 10, serious 5, moderate 3, minor 1, information 0.
Score = 100 × (1 − total weight of failed rules ÷ total weight of applicable rules), rounded to the nearest whole number.
- Applicable rules are those that passed or failed. Rules that do not apply, need review, or could not run are left out.
- The overall score uses the same formula over every rule, and each category's score over its own rules, so categories with heavier rules weigh more in the overall score. Every category the scan ran rules of has a score, or "no score" when none of its rules with weight applied.
- A partial score: when a rule could not run, the report says the score is partial, since it counts only what finished.
- No score: when no rule with weight applies, such as a category of information rules alone, there is no score.
- A page the scan did not reach: when the site answers with an error status, such as 403 or 404, or with a bot challenge instead of the page, the page itself was not checked. The rules that run then read robots.txt, the sitemaps and the server’s answer, and say nothing of the page, so the scan has no score, overall or by category, and is partial.
- The rules counted: the report gives how many of the rule set's rules the scan ran (
score.rules). A scan that does not render the page in a browser leaves out the rules that read the rendered page, and a scan that names its rules runs those alone, so its score counts fewer rules. - Comparing: scores compare only within the same major version of the rule set, which the report gives (
rulesetVersion), since adding a rule or changing a weight changes the score; and between scans that ran the same rules.
An example for each weight
On a page where a critical rule (10) and a serious rule (5) passed, and one rule failed:
| Failed severity | Applicable | Failed | Score |
|---|---|---|---|
| Critical | 10 + 5 + 10 = 25 | 10 | 100 × (1 − 10 ÷ 25) = 60 |
| Serious | 10 + 5 + 5 = 20 | 5 | 100 × (1 − 5 ÷ 20) = 75 |
| Moderate | 10 + 5 + 3 = 18 | 3 | 100 × (1 − 3 ÷ 18) = 83.3, so 83 |
| Minor | 10 + 5 + 1 = 16 | 1 | 100 × (1 − 1 ÷ 16) = 93.75, so 94 |
| Information | 10 + 5 + 0 = 15 | 0 | 100 |
The same examples are tests in packages/scoring, so the formula written here and the one computed cannot differ.
The limits of each source
- The page's HTML as it arrived: it does not see what scripts add after loading. The rules that need that read the rendered page.
- Browsers: each engine at a fixed version the report gives, with one phone-sized window (390 × 844), a fixed language and time zone, and one waiting policy. WebKit on Linux is not Safari: it is Safari's engine, but system fonts and drawing differ.
- What pages send: in the render, every request that is not
GETorHEADis refused before it leaves the browser, wherever it goes, even to the page's own site, and every WebSocket is closed before it opens. What the route that refuses requests does not see is not left to the page: no web worker starts, because what a worker opens, a WebSocket and the requests of a service worker it registers, does not pass it;window.openopens no window;fetchLaterdoes not exist; a form the page submits is not submitted, so that the browser goes on loading it; the handlers a page sets for its own dismissal (pagehide,unload) do not run; and reports of a page's content security policy (CSP) to an address the page names are not sent. So a page that fetches its content with aPOSTrequest, as a GraphQL interface does, or builds it in a worker, is measured without that content, and a tool that reports a visit byPOSTorsendBeacondoes not report this one.GETandHEADrequests go out as a link or an image would, and their address can carry data, so each browser has a limit on requests, on data and on hosts. The report says what was refused, in its count of refused requests and in a notice. - Fonts: fallback fonts differ between devices, so what our server draws when a site's font lacks a character may differ from a visitor's device. So the Docker image is built with set fonts, and reference reports are generated inside it.
- Stylesheets and fonts: a file is read only when its size is known before reading it, within fixed limits. Firefox and WebKit hide the size of another site's stylesheet, so there it is not read, and the rules that need it do not judge it.
- axe-core: a curated set of its rules, each with Arabic copy and test fixtures. A bug makes axe skip the contrast of Arabic text; we correct for it before running it.
- The rules' own limits: every number in a rule, such as the days before a certificate expires or the savings compression must bring, is on the rule's page with its source.
- Real-user data (CrUX): from the Chrome UX Report, with an API key: each metric's 75th percentile on phones over the last 28 days, for the page's URL, or its whole site when CrUX has none for the page, and the report says which. It counts visits in Chrome alone, by users who share usage statistics and sync their browsing history; Chrome on iPhone and other browsers are not counted. Many sites with fewer visits have no data, so its rules do not apply to them. The page's URL is sent to Google, unless the page is on a local or private address.
- DNS records: we read the records of the page's organizational domain, as RFC 7489 defines it from the Public Suffix List, not a subdomain's own. A site a platform gives its customers, such as
user.github.io,shop.myshopify.comorsite.vercel.app, is not asked about and the two rules do not apply to it, since the DNS of that zone is the platform's and not the site owner's. A DMARC record must have aptag with the valuenone,quarantineorreject, or the rule fails. The records of one name are put in a fixed order before the rules read them. A question that gets no answer in time, or an error from the DNS server, says nothing of the records, so we do not judge by it. - Text written by scripts: we compare the Arabic words a browser drew, in the first 200 visible elements with Arabic text and the first 200 characters of each, with all the text of the body of the HTML the scan received, hidden text and
<noscript>and<template>included. A browser is judged only when it drew at least 20 Arabic words, a number of our own for this rule. The page fails when more than half of the words are not in the HTML. The scan asks for the HTML with no language preference while the browsers ask for Arabic, so a site that picks the language of its HTML on the server can look as if scripts wrote its text. - The page's links: we ask for links to the page's own origin alone, that is the same scheme, name and port, the first 50 of them, and see none that scripts add. We follow none of their redirects. A link that gets no answer, or answers
401,403,407,429or503, as sites answer a visitor they take for a bot, is not judged, and after the first429we ask for no more links; the report says how many were not checked. - Lighthouse: version 13 in the render's own browser (Playwright's headless Chromium), on an emulated phone with simulated throttling, and the five metrics its score weighs. They vary from run to run, so they show as information: never findings, and never part of the score. Lighthouse runs only in the command-line tool, on the machine of whoever asks for it with
--lab, and its browser is not yet held to the refusal of requests that send data described above.