Skip to content

robots.txt keeps Googlebot from the page's scripts or styles

robots-blocks-resourcesmoderate
On this page

Why it matters

  • Google renders a page as a browser does, with its scripts and stylesheets. Its JavaScript SEO guide says: "Google Search won't render JavaScript from blocked files".
  • Its robots.txt guide allows blocking files a page can do without, and adds: if their absence would "make the page harder for Google's crawler to understand the page, don't block them".
  • A blocked stylesheet can make a phone page look like a desktop page to Google, and a blocked script can hide the text or links it writes.

Example

Wrong
User-agent: Googlebot
Disallow: /assets/
Right
User-agent: Googlebot
Allow: /

How to fix

Allow Googlebot the folders the page's scripts and styles come from:

robots-txt
User-agent: Googlebot
Allow: /
  • Old rules such as Disallow: /assets/, /wp-includes/ or /js/ are a common cause; remove them, or add an Allow line for the files the page needs.
  • Blocking a file the page does not need, such as an analytics script, is allowed; this check names each file so you can tell.

How we detect

  1. We render the page and list the scripts and stylesheets it loads from its own address (the same scheme, host and port), in each engine.
  2. We test each against the site's robots.txt as Google reads it, for Googlebot: the groups that name it, else the * group; the longest matching rule wins.
  3. Files on other hosts have their own robots.txt and are not judged, nor are images and fonts, nor files that answered an error (resources-broken reports those).