SEMRush is a tool which you can use to crawl your website to analyse to find technical issues with what’s wrong on the website.
SEMRush Crawler
Also, SEMRush offers a domain metric called “Authority Score” which analyses a websites backlinks. Its measured using, page score, domain score, trust score as well as some other data to give a score out of 100. The higher the score the more likely that the site is strong, but in order for it to find these links it has to crawl the web like Google.
Just to be clear – their crawler doesn’t crawl as much as Google so it’s unlikely to pick up on every link that you have. In fact, no one crawler would, in theory, be able to identify every link you had – its almost impossible to crawl every single webpage, every single day looking for new links.
In order for SEMRush to calculate this score they need to crawl the web looking for backlinks, so even if you don’t use SEMRush you may still see that their bot has crawled your site by looking in your Server Logs.
SEMRush User Agent
I have tried to list below the main user agents they will use when crawling your website.
- Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)
- Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)
- Mozilla/5.0 (compatible; SemrushBot/3~bl; +http://www.semrush.com/bot.html)
- Mozilla/5.0 (compatible; SemrushBot-BA; +http://www.semrush.com/bot.html)
- Mozilla/5.0 (compatible; SemrushBot/1.0~bm; +http://www.semrush.com/bot.html)
- Mozilla/5.0 (compatible; SemrushBot/1.0~bm; +http://www.semrush.com/bot.html)
These are the most recent User Agents I have seen and will keep updating this article to include future User Agents that SEMRush may use.
The top SEMRush bot – is their mobile bot that is used to crawl your website.
In fact the top two bots, are from my understanding is the bots that they use as part of their audit feature.
SEMRush IP Addresses:
It would take a proper geek to do this, but it can be done – you can mask a User Agent and make it look like a service/company is crawling your site when they are not.
The one thing you can’t mask is the IP address.
SEMRush IPs are:
- 46.229.173.66
- 46.229.173.67
- 213.174.147.83
- 192.243.56.76
If the IP’s don’t match the above when analysing,its likely somebody editing their setting and acting like they are SEMRush.
SEMRush might also change their IP range over time and we will try and keep this as upto date as possible.
SEMRush Bot Attack:
Someone once asked me if you could use Web-based Crawling tools like a DDOS attack service and while in theory, my answer was yes, it’s very unlikely to be possible.
Firstly these tools only crawl one url and move on, not repeatedly hit the same URL and even if you created Multiple “Projects” and used all the different software providers it would still require you to be extremely lucky and wouldn’t last very long.
That all said, if you have a poor hosting solution and the additional traffic it could be possible to bring down your site, but you would need to be a very poor set up for that to be possible.
Blocking SEMRush Bot:
A few months ago I read an article from a hosting company which said that all SEO tool bots should be blocked as it uses bandwidth and they don’t add any value. While I understood what he was arguing I didn’t agree to it, without third-party metrics and analysis judging a website would be difficult.
There are many reasons you may want to block certain bots from crawling your site. You might have a hidden PBN network that you don’t want your competition to pick up on a report to the Google Police or you could generally be looking to reduce your server costs.
In this article, The best ways to block bots from crawling your site – I have explained the different methods you can use to block certain bots, but to block SEMRush via the Robots file you just need to add:
Please note, while you add this to your site robots file – it can take a while for it to take effect as SEMRush needs to crawl your robots file to see the changes. While Google does crawl this page frequently I have noticed SEMRush bot does it a lot less frequently.
If you are happy for their link scoring bot to crawl your site to find new links etc but want to stop your competitors or others from crawling your site to find technical issues then you can add the following to your robots file.
Try SEMRush
SEMRush Offer a 14 Days Free Trial – simply visit SEMRush via this link to get the free trial
Hey Andrew,
I didn’t read the article you mention about blocking SEO bots but here is a thought. If one doesn’t subscribe to a set of SEO tools then perhaps NOT blocking their bots only goes to assist the competition who may use those tools and glean information about you which you are not benefiting from yourself. Of course most of the tools offer some free information.
All the best,
Jon