Skip to content

Part of the page lies past what Googlebot fetches

html-over-fetch-limitserious
On this page

Why it matters

  • Google's documentation says Googlebot crawls the first 2 MB of an HTML file for Google Search, and that the limit applies to the uncompressed data. What comes after is not fetched, so it is not indexed.
  • Pages get this large from inline data, not from text: a framework's state serialized into the HTML, inline images in base64, a huge menu, inline SVG sprites. When that comes first, the canonical, the structured data and the article itself can fall past the cut.
  • A large page whose important parts come first loses only its tail. That is why this check reports what is past the cut, not the size.

Example

Wrong
<script>window.__STATE__ = "<!-- PAD -->"</script>
<link rel="canonical" href="/" />
Right
<p>عطور شرقية وعود وبخور، تُشحن الطلبات خلال يومين إلى كل المدن.</p>
<script>window.__STATE__ = "<!-- PAD -->"</script>

How to fix

Put what matters first, and move bulk out of the HTML:

html
<head>
  <title>عطور الواحة | متجر الواحة</title>
  <link rel="canonical" href="https://shop.example/ar/" />
  <script type="application/ld+json">…</script>
</head>
<body>
  <main>…</main>
  <script src="/state.js" defer></script>
</body>
  • Load a framework's state from a file, or trim it to what the page needs.
  • Serve images as files, not as data: addresses in the HTML.
  • Keep <head> metadata near the top, before any large inline script or style.
  • Compression does not help: the limit counts the uncompressed bytes.

How we detect

  1. We take the body as the server sent it, decompressed, and find where its first 2 MB end: 2 × 1024 × 1024 bytes, the larger reading of 2 MB, counted per character in the page's encoding (two bytes for most Arabic letters in UTF-8, one in windows-1256).
  2. We report the title, the description and robots meta tags, the canonical and hreflang links, JSON-LD blocks and the first <h1> that start past that point, and the visible text when more than half of its letters are past it.
  3. A page longer than the limit with nothing of these past it is not reported.

This is Googlebot's limit for Google Search; other crawlers have their own. Each resource the page loads has a limit of its own, not read here.