So have you been looking through your log files and notice that our user=agent has crawled your site.
Its either myself or The Server Log Kid – ok it’s probably him.
We don’t tend to crawl sites which aren’t our clients, so it’s unlikely for you to see it unless we have done some work in the past, but see below for a couple of reason we might be crawling you if you haven’t specifically reached out.
The UserAgent: “OnpageRocks https://onpage.rocks/user-agent”
Blocking Our User Agent
The bot is instructed to respect robots.txt file for:
So if you prefer we didn’t crawl your site, please update your robots file, but who can say no to the server log kid crawling your site.
We haven't employed you
A question I get a lot is that “we haven’t employed you why are you crawling our site”.
First of all, well done on analysing your server logs, very few companies actively monitor their logs so it’s great that you are.
Usually, it’s one of two reasons, we do a lot of what is known as “white label” auditing, whereby other agencies and freelancers hire me to do the work.
The other reason is some of bigger clients actually hire us to audit their competition so they can understand their weaknesses from a tech point of view to see where they have a competitive advantage.
Great articles from the Blog
Unsurprisingly, as they never do anything by half, there are quite a few different Google user agents. But, just how many there are might surprise
We all know that robots were created to make life easier (or to make us lazier!), so it stands to reason that the robots.txt file
When it comes to domain names www, short for world wide web, has been around for some considerable time – as I’m sure you’re aware