Unsurprisingly, as they never do anything by half, there are quite a few different Google user agents. But, just how many there are might surprise you.
I have been analysing server logs since around 2013, and, because I like to keep track, I have kept a record of all the different User-Agents I have come across.
Currently, in the database, there are 53 different Google bots.
What might also surprise you is the variety of user agents, and what they all do….
What User Agents Are Looking At My Site?
Some of the most popular user-agents include:
- Google AdsBot
- Google Feedfetcher
- Google Lighthouse
- Google Mobile AdsBot
- Google Read Aloud
- Google Site Verifier Bot 1.0
- Google Structured Data Testing Tool
- Google’s Media Partners system (AdSense)
- Googlebot
- Googlebot 2.1
However, there is a problem (isn’t there always?), and that’s that it is quite easy to mask a user-agent. In fact, most web-based crawlers and indeed desktop-based crawlers give you the option to change the user agent to Googlebot.
Is It Google, Or Not?
The one thing they can’t change is the IP address. So, if you really want to analyse specifically what Google is doing on your site, you’ll need to do a reverse IP lookup on the IP address and see if it’s REALLY from GoogleHQ.
In an ideal world (for me, anyway) these tools would slightly modify the user agent to something like ” Googlebot 2.1- Tool name” so it still followed the rules for Googlebot in the robots.txt file but didn’t mess up server log auditing. We live in hope that this may happen in the future, but for the time being, we’ll do it the hard way.