Fetch as Google: Seeing Your Website as Google Sees It

Fetch as Google: Seeing Your Website as Google Sees It

Users of webmaster tools can see how Google crawls and renders their URLs with the assistance of the “Fetch as Google” tool. Working the same as Googlebot, the tool enables users to check for issues such as blocked resources and to verify that Google is able to render dynamic content correctly to play a role in search results.

Why Is This Important?

In ranking pages, Google considers non-textual content and visual layout as well as text content. The search engine uses this information to determine quality and value compared against other sites. Fetching as Google allows you to see your site just as Google sees it. By checking that Googlebot can crawl your site, you can ensure that your webpages will appear in organic search results for the correct search terms.

This post will teach you how to use the Fetch as Google tool.

A Step-by-Step Guide to Running the Fetch

Fetch as Google: Seeing Your Website as Google Sees It

Fetching URLs

You will need to fetch every URL on your site separately. Enter only the path component into the text box or add no information at all to fetch the root page. As the root is already included (and cannot be changed) you will be unable to fetch for anything other than the current site.

Googlebot type

For fetching, you can choose between several types of Googlebots. All fall into one of two categories: desktop and mobile. Desktop is the default and is further split into five types:

  1. Googlebot crawler for websites and news
  2. Googlebot Images crawler
  3. Googlebot Video crawler
  4. Google AdSense crawler
  5. Google AdsBot crawler.

For mobile, you can choose between the current and upcoming Google Smartphone crawler.


“Fetch” or “Fetch and Render”

Next you need to choose whether you want to just fetch or both fetch and render.

The former will fetch the URL and will present you with an HTTP response. Fetch is ideal for a fast check in the case you suspect your website has malware, you are experiencing a network connectivity or security problem, or you just want to see if your URL will fail or succeed.

The latter does all that Fetch does and additionally renders your page to the platform you choose — desktop or smartphone — and runs all the resources, including images and scripts. This will allow you to compare how a user sees a page with how Googlebot sees the page.

Blocked Resources

A fetch and render may uncover blocked resources on some pages: JavaScript, CSS, and image files can all be blocked to Googlebot. If something is blocked, Google may be unable to render and index the page properly. To determine whether any of your resources are blocked, you need to look at your Blocked Resources Report. In this report, Google presents you with the blocked resources it believes are within your capacity to change.

Finding blocked resources

Start at the report landing page. Here you will find the list of hosts providing resources that are blocked according to robots.txt rules. Within the list, there will likely be a combination of resources hosted by your own site and some hosted by other sites.

By clicking on each of the hosts, you can see what blocked resources they are responsible for. You will also be able to find out how many pages on the site are impacted by every blocked resource. Click on the resource to view which pages are affected.

Unblocking the resources

Within the same report, you can find out how to unblock the resources. It is best to start with resources from hosts you own, as you have the most control over these. Some will not be worth the time to unblock, such as resources that track pixels and counters. Leave these as they are and move on to the more essential resources, especially those that may impact content and layout.

By selecting the page where the blocked resource is present, you can see if the resource is impacting appearances. In the case it is, use the pop-up instructions to update the file. After every change you make, you should fetch and render again to check that the page now appears correctly.

Once you have enabled Googlebot to access the most important resources from the hosts you own, repeat the task with those from hosts you do not own. Once you have located blocked resources that may have a significant impact on your site, you will need to either contact the webmaster of the host site or remove the resource entirely.

Having trouble with Fetch as Google or other webmaster tools? Head over to Boostability for more training materials or a free consultation!