Skip to content

ArablyzerBot, Arablyzer’s bot

The bot that fetches your page for a check: what it does, what it never does, how to block it.

On this page

How to recognise it

Every request it sends outside a browser has this user agent:

ArablyzerBot/1.0 (+https://arablyzer.com/bot)

When it renders the page in browsers, each browser keeps its own user agent and adds this token at its end:

… ArablyzerBot/1.0 (+https://arablyzer.com/bot)

When it visits your site

  • When someone asks to check a page of your site, on a tool’s page or with the full scan. It never visits on its own, and it does not follow links to other pages.
  • It checks the page asked for alone: if that page redirects, it follows, and opens no other page.

What it fetches

  1. robots.txt first, to know whether you block it, following at most 5 redirects and reading its first 500 KB.
  2. The page itself, following at most 10 redirects.
  3. When the check needs the domain’s mail records: two TXT lookups, one for the page’s domain and one for its _dmarc name, as DNS over HTTPS to Cloudflare’s resolver (https://cloudflare-dns.com/dns-query) or to the resolver the service is configured with, through the same egress proxy. The domain’s name goes to that resolver, and nothing else of the page.
  4. When the check needs the page’s links, as the broken-link check does: a HEAD request to each of the first 50 links on the page to your own site, and a GET where it answers an error or the connection fails, following no redirect and reading no content, and skipping the paths robots.txt disallows for ArablyzerBot or for every crawler with User-agent: *. After the first 429 it asks for no more links.
  5. When the check reads your sitemaps: the first 3 your robots.txt names, or /sitemap.xml when it names none, reading the first 25 MB of each and following at most 10 redirects, and never the sitemaps a sitemap index lists. It waits no more than 10 seconds on the network for all of them.
  6. A sitemap your robots.txt names may be on another host: the bot reads that host’s robots.txt first, does not fetch what it asks ArablyzerBot, or every crawler with User-agent: *, to leave alone, and follows a redirect to another host only where that host’s robots.txt allows it.
  7. When the check renders the page: it opens it in Chromium, Firefox and WebKit, in a phone window 390 wide and 844 high, and each browser loads what the page loads for a visitor (stylesheets, fonts, images, scripts), at most 300 requests and 25 MB per browser, to at most 50 different hosts.
  8. Every request goes through one egress proxy that refuses private and local addresses.

What it never does

  • It never submits a form, types into a field, or presses a button.
  • It sends nothing your page asks it to send: its browsers refuse every request that is not GET or HEAD, and every WebSocket, wherever it goes, so no form, sendBeacon or POST request your page makes leaves a check, even to your own site. An analytics tag that reports a visit by POST or by sendBeacon does not report this one.
  • It never logs into an account.
  • It never gets past a CAPTCHA or any bot protection: if your site answers with a challenge, the report says the site blocked the check, and if only a browser is answered with one, that browser’s render stops there.

How to block it

Add a group naming it to your site’s robots.txt, and no address of your site is checked:

User-agent: ArablyzerBot
Disallow: /

Or block only what you want blocked, and the rest stays open to checks:

User-agent: ArablyzerBot
Disallow: /account/
Disallow: /checkout/
  • ArablyzerBot reads robots.txt at the start of every check, so a block holds from the next check.
  • When a page is blocked, the check stops before fetching it, and the report says the site asked not to be checked.
  • When a page redirects to another site, it reads that site’s robots.txt too before going on.
  • A User-agent: * group alone does not block a check someone asks for: that is a visit to one page, not a crawl of your site. The requests for the page’s links are not that visit, and the * group applies to them: the check asks for no path it disallows.
  • When robots.txt cannot be read, as when it answers with a server error, the check goes on.