So SEMRush Crawling tool is taking on Screaming Frog Crawling tool to determine which tool is the best for crawling your website to find technical issues.
While Screaming Frog is from Britain, a small town in Oxford, SEMRush is a more global operation. There is Head Office is based in Boston in the United States, but its there Russian office playing this game, after all its the Champions League and that is made up of European teams
We do a lot of website auditing here and we have tried a wide variety of software and tools to help us make the job easier.
In this article, I am going to show you the key differences between Screaming Frog and SEMRush to try and help you determine which one is the best for you.
We are going to compare them on several key points which we think are crucial.
Before we get into the specifics, I just wanted to highlight the key differences, Screaming Frog is a desktop based application which you download and run locally on your machine, SEMRush Audit is a cloud-based tool. This is a key difference between the two bits of software.
There are two versions of Screaming Frog, a free version which limits some features and only allows you to crawl 500 pages and a paid plan which has a yearly fee of £149. This analysis assumes you have the paid version unless otherwise stated.
We also realise SEMRush offers a lot more than just auditing but for the purpose of this analysis, we are just comparing their auditing solutions.
The Size of Your Website:
This one is important. If your site is sub 10,000 pages then it’s not really an issue for either option. However depending on the specs of your machine then Screaming Frog may struggle to run larger sites.
Winner
Draw (small sites, Screaming Frog, large Sites SEMRush)
The Number of Websites:
If you run/manage one website this isn’t an issue, however, if you manage a few small clients, maybe have a few small affiliate sites then it can be an issue.
While SEMRush does give you a decent amount of pages to crawl even on their basic plan (100,000 pages) you can only have 5 projects meaning you can only crawl 5 different websites per month.
That does increase to 50 projects and 200 projects with their different plans. With their top Tier plan as well as being able to crawl 200 different sites, you can crawl 1 million URLs.
Screaming Frog, on the other hand, doesn’t limit the number of sites you can audit or how many pages you can crawl. It’s down to your machine.
The only other limit is you have to set the next crawl going, which means managing the process.
Winner
Screaming Frog (assuming you have quite a few small sites to manage).
Scheduling of Crawls:
When creating a new project this is part of the setup process, how frequently would you like to crawl this site. This ranges from once, once a day or weekly. It can easily be changed in the setting if after setting it up you change your mind.
Screaming Frog – it can be done quite easy witht Screaming Frog version 10+ however it does still require your machine to be on.
Winner
For ease of use, this has to go to SEMRush, it can be done in Screaming Frog as mentioned above.
Finding Errors:
1 goal each here for the two teams, they both have their positives.
Let’s be honest here, none of us really want to be crawling our sites to find errors (there are much more exciting things to be doing in life) but we need to do it. Whether its a stray WordPress plugin making changes by accident, a developer releasing something without testing, or you editing the site after a few beers and not checking your own work. Errors occur for a number of reason. Quickly finding and fixing these issues will help both with SEO and UX.
They both report on about 90% of the same issues, Screaming Frog I find seems to report more of the issues sometimes that SEMRush can sometimes miss, that’s why I use it for the big weekly analysis.
That said SEMRush does some additional checks that Screaming Frog currently doesn’t like checking Sitemaps for errors, alerting if Robot.txt has changed.
SEMRush breaks the issues they find down into three sections, “Errors”, “Warnings” and “Notices” depending on how much the issue affects your ability rank, determines where it falls. Meaning you know which issues to focus on first.
Winner
Draw, while Screaming Frog does tend to find more of the known issues (not always, but there are occasions) SEMRush does do some additional checks which is why these to draw. Using the two tools combined is the best option.
Reporting:
If the reason we do the analysis is to find the errors, reporting on the issues is just as important.
SEMRush being a cloud-based tool has a very nice dashboard which highlights all the issues – you can see the data if you want, but it makes it very nice and simple at a quick glance.
Screaming Frog, on the other hand, has no reporting dashboard, we created a free report using Google Sheets and DataStudio that you can use. The Screaming Frog Dashboard is easy to use, but as it’s not officially from Screaming Frog we won’t count it.
You could also create your own dashboard using Excel or Data Studio.
SEMRush also offers you a “Score” out of 100. So you can see how well you are doing.
Winner
SEMRush – has to win, being cloud-based does give it the advantage, butt it is possible to do it on your own using Screaming Frog
Historical Comparison:
As SEO’s and webmasters, we tend to like to see progress and how well we are doing.
SEMRush – give you the ability to track progress comparing different crawls at the URL level, but you can also track the progress of specific issues to see whether they are improving, getting worse or staying the same.
Screaming Frog doesn’t offer this feature.
Winner
By default, it has to be SEMRush.
Crawl a list of URLs:
This is a very useful feature, whether you doing server log analysis and want to check the current status of a list of URLs or you want to see where specific URLs are redirecting too. In fact, this is a feature we use very frequently.
Screaming Frog has three Mode, “Spider”, “List” or “Serp” – they have a very specific feature for this.
SEMRush – has just released this feature and while it’s a great addition it does require the use of an additional Project
Winner
Screaming Frog, while SEMRush has just launched this feature, it’s just easier in Screaming Frog at the moment.
Costs:
Costs are important. It’s likely you have a budget or if it’s your own business then you will be wanting to make sure you invest your cash in the right area’s.
Screaming Frog – You buy a licence which lasts a year and can be used on two devices. The price is £149 for the year. If you have a very small site (less than 500 pages) you could use the free version of SF.
SEMRush being a cloud-based service is much more expensive and has a few tier prices. Starting at just $99 a month to $399 a month – however, you can pay for a year and save up to $800. If you want additional users there is also an additional cost.
SEMRush does offer a free 7 Day trial so you can see what their platform is like. We have partnered with SEMRush to offer you a 14 Day Trial
Winner
When it comes to price Screaming Frog is the clear winner here. The price is a bit of a steal if I’m honest. It’s a great piece of software.
Which one is the best?
This isn’t about me sitting on a fence and not picking a winner – I like both pieces of software, I use both pieces of software.
I do large weekly / monthly audits using Screaming Frog to have all the raw data and to be able to truly deep dive, but then I have SEMRush which runs daily, while it doesn’t do as a deep analysis it gives me a top line view and a score so I can quickly see if anything has broken.
Some of our clients pay us to monitor their sites and its impossible for us to monitor every site daily. However, we do big audits once a week and let SEMRush pick up on daily auditing. This has allowed us to scale but also offer a very high service.
Luckily SEMRush offers a Free 14 Day Trail via this link and if your site is less than 500 pages you can try Screaming Frog for free and make up your own mind.
Let us know in the comments below which one you think is best?
Video Review
The Tools
