Onpage Rocks User-Agent

Onpage Rocks


So have you been looking through your log files and notice that our user=agent has crawled your site.

Its either myself or The Server Log Kid – ok it’s probably him.

We don’t tend to crawl sites which aren’t our clients, so it’s unlikely for you to see it unless we have done some work in the past, but see below for a couple of reason we might be crawling you if you haven’t specifically reached out.

The Useragent

The UserAgent: “OnpageRocks https://onpage.rocks/user-agent”

Blocking Our User Agent

The bot is instructed to respect robots.txt file for:


So if you prefer we didn’t crawl your site, please update your robots file, but who can say no to the server log kid crawling your site.

We haven't employed you

A question I get a lot is that “we haven’t employed you why are you crawling our site”.

First of all, well done on analysing your server logs, very few companies actively monitor their logs so it’s great that you are.

Usually, it’s one of two reasons, we do a lot of what is known as “white label” auditing, whereby other agencies and freelancers hire me to do the work. 

The other reason is some of bigger clients actually hire us to audit their competition so they can understand their weaknesses from a tech point of view to see where they have a competitive advantage. 


Great articles from the Blog


Google Bot User Agents

Unsurprisingly, as they never do anything by half, there are quite a few different Google user agents. But, just how many there are might surprise

Subscribe to my newsletter

Want to learn more about technical SEO and Server Log Analysis, sign up below and I will teach you.

Each week I will send you the most helpful articles delivered straight into your inbox as well as sharing some very useful tips and tricks to improve your technical SEO skills to Ninja level.

* indicates required