Unsurprisingly, as they never do anything by half, there are quite a few different Google user agents. But, just how many there are might surprise you.

I have been analysing server logs since around 2013, and, because I like to keep track, I have kept a record of all the different User-Agents I have come across. 

Currently, in the database, there are 53 different Google bots

What might also surprise you is the variety of user agents, and what they all do….

What User Agents Are Looking At My Site?

Some of the most popular user-agents include:

  1. Google AdsBot  
  2.  Google Feedfetcher  
  3.  Google Lighthouse  
  4.  Google Mobile AdsBot  
  5.  Google Read Aloud  
  6.  Google Site Verifier Bot 1.0  
  7.  Google Structured Data Testing Tool  
  8.  Google’s Media Partners system (AdSense)  
  9.  Googlebot  
  10.  Googlebot 2.1

However, there is a problem (isn’t there always?), and that’s that it is quite easy to mask a user-agent. In fact, most web-based crawlers and indeed desktop-based crawlers give you the option to change the user agent to Googlebot. 

Is It Google, Or Not?

The one thing they can’t change is the IP address. So, if you really want to analyse specifically what Google is doing on your site, you’ll need to do a reverse IP lookup on the IP address and see if it’s REALLY from GoogleHQ.

In an ideal world (for me, anyway) these tools would slightly modify the user agent to something like ” Googlebot 2.1- Tool name” so it still followed the rules for Googlebot in the robots.txt file but didn’t mess up server log auditing. We live in hope that this may happen in the future, but for the time being, we’ll do it the hard way.

Share:

Share on facebook
Facebook
Share on twitter
Twitter
Share on pinterest
Pinterest
Share on linkedin
LinkedIn

Leave a Comment

Your email address will not be published. Required fields are marked *

On Key

More Great Articles

Technical Seo Demystified - Martin Splitt
Conferences
Andy Halliday

Technical Seo Demystified: Martin Splitt

https://www.youtube.com/watch?v=X05ypQmLXc8Video can’t be loaded because JavaScript is disabled: Martin Splitt – Technical SEO Demystified | AngualrUP 2019 (https://www.youtube.com/watch?v=X05ypQmLXc8) This is a great talk from one

Read More »

Subscribe to my newsletter

Want to learn more about technical SEO and Server Log Analysis, sign up below and I will teach you.Each week I will send you the most helpful articles delivered straight into your inbox as well as sharing some very useful tips and tricks to improve your technical SEO skills to Ninja level.

Subscribe to our mailing list

* indicates required