Why it matters
- Google renders a page as a browser does, with its scripts and stylesheets. Its JavaScript SEO guide says: "Google Search won't render JavaScript from blocked files".
- Its robots.txt guide allows blocking files a page can do without, and adds: if their absence would "make the page harder for Google's crawler to understand the page, don't block them".
- A blocked stylesheet can make a phone page look like a desktop page to Google, and a blocked script can hide the text or links it writes.
Example
User-agent: Googlebot Disallow: /assets/
User-agent: Googlebot Allow: /
How to fix
Allow Googlebot the folders the page's scripts and styles come from:
robots-txt
User-agent: Googlebot
Allow: /- Old rules such as
Disallow: /assets/,/wp-includes/or/js/are a common cause; remove them, or add anAllowline for the files the page needs. - Blocking a file the page does not need, such as an analytics script, is allowed; this check names each file so you can tell.
How we detect
- We render the page and list the scripts and stylesheets it loads from its own address (the same scheme, host and port), in each engine.
- We test each against the site's robots.txt as Google reads it, for Googlebot: the groups that name it, else the
*group; the longest matching rule wins. - Files on other hosts have their own robots.txt and are not judged, nor are images and fonts, nor files that answered an error (
resources-brokenreports those).