Unsurprisingly, as they never do anything by half, there are quite a few different Google user agents. But, just how many there are might surprise you.

I have been analysing server logs since around 2013, and, because I like to keep track, I have kept a record of all the different User-Agents I have come across. 

Currently, in the database, there are 53 different Google bots

What might also surprise you is the variety of user agents, and what they all do….

What User Agents Are Looking At My Site?

Some of the most popular user-agents include:

  1. Google AdsBot  
  2.   Google Feedfetcher  
  3.   Google Lighthouse  
  4.   Google Mobile AdsBot  
  5.   Google Read Aloud  
  6.   Google Site Verifier Bot 1.0  
  7.   Google Structured Data Testing Tool  
  8.   Google’s Media Partners system (AdSense)  
  9.   Googlebot  
  10.   Googlebot 2.1

However, there is a problem (isn’t there always?), and that’s that it is quite easy to mask a user-agent. In fact, most web-based crawlers and indeed desktop-based crawlers give you the option to change the user agent to Googlebot. 

Is It Google, Or Not?

The one thing they can’t change is the IP address. So, if you really want to analyse specifically what Google is doing on your site, you’ll need to do a reverse IP lookup on the IP address and see if it’s REALLY from GoogleHQ.

In an ideal world (for me, anyway) these tools would slightly modify the user agent to something like ” Googlebot 2.1- Tool name” so it still followed the rules for Googlebot in the robots.txt file but didn’t mess up server log auditing. We live in hope that this may happen in the future, but for the time being, we’ll do it the hard way.


Share on facebook
Share on twitter
Share on pinterest
Share on linkedin

Leave a Comment

Your email address will not be published. Required fields are marked *

On Key

More Great Articles

Andy Halliday

Technical SEO Tips

So in the run-up to Christmas, I did 24 tips in 24 days – while it might not have tasted as good as a chocolate

Read More »
How to find Orphan pages

How to find Orphan Pages

Finding Orphan pages are super important and can be very easy to find. In this video, we are going to show you how to quickly

Read More »
SEMRush Bot

SEMRush Bot

SEMRush is a tool which you can use to crawl your website to analyse to find technical issues with what’s wrong on the website. SEMRush

Read More »

Subscribe to my newsletter

Want to learn more about technical SEO and Server Log Analysis, sign up below and I will teach you.

Each week I will send you the most helpful articles delivered straight into your inbox as well as sharing some very useful tips and tricks to improve your technical SEO skills to Ninja level.

* indicates required