Users of webmaster tools can see how Google crawls and renders their URLs with the assistance of the “Fetch as Google” tool. Working the same as Googlebot, the tool enables users to check for issues such as blocked resources and to verify that Google is able to render dynamic content correctly to play a role in search results.
Why Is This Important?
In ranking pages, Google considers non-textual content and visual layout as well as text content. The search engine uses this information to determine quality and value compared against other sites. Fetching as Google allows you to see your site just as Google sees it. By checking that Googlebot can crawl your site, you can ensure that your webpages will appear in organic search results for the correct search terms.
This post will teach you how to use the Fetch as Google tool.
A Step-by-Step Guide to Running the Fetch
You will need to fetch every URL on your site separately. Enter only the path component into the text box or add no information at all to fetch the root page. As the root is already included (and cannot be changed) you will be unable to fetch for anything other than the current site.
For fetching, you can choose between several types of Googlebots. All fall into one of two categories: desktop and mobile. Desktop is the default and is further split into five types:
- Googlebot crawler for websites and news
- Googlebot Images crawler
- Googlebot Video crawler
- Google AdSense crawler
- Google AdsBot crawler.
For mobile, you can choose between the current and upcoming Google Smartphone crawler.
“Fetch” or “Fetch and Render”
Next you need to choose whether you want to just fetch or both fetch and render.
The former will fetch the URL and will present you with an HTTP response. Fetch is ideal for a fast check in the case you suspect your website has malware, you are experiencing a network connectivity or security problem, or you just want to see if your URL will fail or succeed.
The latter does all that Fetch does and additionally renders your page to the platform you choose — desktop or smartphone — and runs all the resources, including images and scripts. This will allow you to compare how a user sees a page with how Googlebot sees the page.
Finding blocked resources
Start at the report landing page. Here you will find the list of hosts providing resources that are blocked according to robots.txt rules. Within the list, there will likely be a combination of resources hosted by your own site and some hosted by other sites.
By clicking on each of the hosts, you can see what blocked resources they are responsible for. You will also be able to find out how many pages on the site are impacted by every blocked resource. Click on the resource to view which pages are affected.
Unblocking the resources
Within the same report, you can find out how to unblock the resources. It is best to start with resources from hosts you own, as you have the most control over these. Some will not be worth the time to unblock, such as resources that track pixels and counters. Leave these as they are and move on to the more essential resources, especially those that may impact content and layout.
By selecting the page where the blocked resource is present, you can see if the resource is impacting appearances. In the case it is, use the pop-up instructions to update the file. After every change you make, you should fetch and render again to check that the page now appears correctly.
Once you have enabled Googlebot to access the most important resources from the hosts you own, repeat the task with those from hosts you do not own. Once you have located blocked resources that may have a significant impact on your site, you will need to either contact the webmaster of the host site or remove the resource entirely.
Having trouble with Fetch as Google or other webmaster tools? Head over to Boostability for more training materials or a free consultation!