SEMRush is a tool which you can use to crawl your website to analyse to find technical issues with what’s wrong on the website.

SEMRush Crawler

Also, SEMRush offers a domain metric called “Authority Score” which analyses a websites backlinks. Its measured using, page score, domain score, trust score as well as some other data to give a score out of 100. The higher the score the more likely that the site is strong, but in order for it to find these links it has to crawl the web like Google.

Just to be clear – their crawler doesn’t crawl as much as Google so it’s unlikely to pick up on every link that you have. In fact, no one crawler would, in theory, be able to identify every link you had – its almost impossible to crawl every single webpage, every single day looking for new links.

In order for SEMRush to calculate this score they need to crawl the web looking for backlinks, so even if you don’t use SEMRush you may still see that their bot has crawled your site by looking in your Server Logs.

SEMRush User Agent

I have tried to list below the main user agents they will use when crawling your website.

  • Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)
  • Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)
  • Mozilla/5.0 (compatible; SemrushBot/3~bl; +http://www.semrush.com/bot.html)
  • Mozilla/5.0 (compatible; SemrushBot-BA; +http://www.semrush.com/bot.html)
  • Mozilla/5.0 (compatible; SemrushBot/1.0~bm; +http://www.semrush.com/bot.html)
  • Mozilla/5.0 (compatible; SemrushBot/1.0~bm; +http://www.semrush.com/bot.html)

These are the most recent User Agents I have seen and will keep updating this article to include future User Agents that SEMRush may use.
The top SEMRush bot – is their mobile bot that is used to crawl your website.
In fact the top two bots, are from my understanding is the bots that they use as part of their audit feature.

SEMRush IP Addresses:

It would take a proper geek to do this, but it can be done – you can mask a User Agent and make it look like a service/company is crawling your site when they are not.
The one thing you can’t mask is the IP address.

SEMRush IPs are:

  • 46.229.173.66
  • 46.229.173.67
  • 213.174.147.83
  • 192.243.56.76

If the IP’s don’t match the above when analysing,its likely somebody editing their setting and acting like they are SEMRush.

SEMRush might also change their IP range over time and we will try and keep this as upto date as possible.

SEMRush Bot Attack:

Someone once asked me if you could use Web-based Crawling tools like a DDOS attack service and while in theory, my answer was yes, it’s very unlikely to be possible.

Firstly these tools only crawl one url and move on, not repeatedly hit the same URL and even if you created Multiple “Projects” and used all the different software providers it would still require you to be extremely lucky and wouldn’t last very long.

That all said, if you have a poor hosting solution and the additional traffic it could be possible to bring down your site, but you would need to be a very poor set up for that to be possible.

Blocking SEMRush Bot:

A few months ago I read an article from a hosting company which said that all SEO tool bots should be blocked as it uses bandwidth and they don’t add any value. While I understood what he was arguing I didn’t agree to it, without third-party metrics and analysis judging a website would be difficult.

There are many reasons you may want to block certain bots from crawling your site. You might have a hidden PBN network that you don’t want your competition to pick up on a report to the Google Police or you could generally be looking to reduce your server costs.

In this article, The best ways to block bots from crawling your site – I have explained the different methods you can use to block certain bots, but to block SEMRush via the Robots file you just need to add:

User-agent: SemrushBot
Disallow: /

Please note, while you add this to your site robots file – it can take a while for it to take effect as SEMRush needs to crawl your robots file to see the changes. While Google does crawl this page frequently I have noticed SEMRush bot does it a lot less frequently.

If you are happy for their link scoring bot to crawl your site to find new links etc but want to stop your competitors or others from crawling your site to find technical issues then you can add the following to your robots file.

User-agent: SemrushBot-SA
Disallow: /

Try SEMRush

SEMRush Offer a 14 Days Free Trial – simply visit SEMRush via this link to get the free trial

Share:

Share on facebook
Facebook
Share on twitter
Twitter
Share on pinterest
Pinterest
Share on linkedin
LinkedIn

1 thought on “SEMRush Bot”

  1. Hey Andrew,
    I didn’t read the article you mention about blocking SEO bots but here is a thought. If one doesn’t subscribe to a set of SEO tools then perhaps NOT blocking their bots only goes to assist the competition who may use those tools and glean information about you which you are not benefiting from yourself. Of course most of the tools offer some free information.
    All the best,
    Jon

Leave a Comment

Your email address will not be published. Required fields are marked *

On Key

More Great Articles

General
Andy Halliday

Technical SEO Tips

So in the run-up to Christmas, I did 24 tips in 24 days – while it might not have tasted as good as a chocolate

Read More »
Google bot user agent
Google
Andy

Google Bot User Agents

Unsurprisingly, as they never do anything by half, there are quite a few different Google user agents. But, just how many there are might surprise

Read More »
How to find Orphan pages
Server Log Kid
Andy

How to find Orphan Pages

Finding Orphan pages are super important and can be very easy to find. In this video, we are going to show you how to quickly

Read More »

Subscribe to my newsletter

Want to learn more about technical SEO and Server Log Analysis, sign up below and I will teach you.Each week I will send you the most helpful articles delivered straight into your inbox as well as sharing some very useful tips and tricks to improve your technical SEO skills to Ninja level.

Subscribe to our mailing list

* indicates required