So have you been looking through your log files and notice that our user=agent has crawled your site.
Its either myself or The Server Log Kid – ok it’s probably him.
We don’t tend to crawl sites which aren’t our clients, so it’s unlikely for you to see it unless we have done some work in the past, but see below for a couple of reason we might be crawling you if you haven’t specifically reached out.
The UserAgent: “OnpageRocks https://onpage.rocks/user-agent”
Blocking Our User Agent
The bot is instructed to respect robots.txt file for:
So if you prefer we didn’t crawl your site, please update your robots file, but who can say no to the server log kid crawling your site.
We haven't employed you
A question I get a lot is that “we haven’t employed you why are you crawling our site”.
First of all, well done on analysing your server logs, very few companies actively monitor their logs so it’s great that you are.
Usually, it’s one of two reasons, we do a lot of what is known as “white label” auditing, whereby other agencies and freelancers hire me to do the work.
The other reason is some of bigger clients actually hire us to audit their competition so they can understand their weaknesses from a tech point of view to see where they have a competitive advantage.
Great articles from the Blog
This talk was given by Luke Monaghan at SearchLeeds last year. SearchLeeds is a great northern UK based general-purpose Marketing conference. The event itself is
https://youtu.be/nSUbD_JB5fM This is a great conference, which I plan on attending in 2o20 – but the great thing is in the past the entire conference