This is something that I have dealt with numerous times and can be hard for certain organisations to set up.
Getting log data when using Cloudflare, by default the logs just show the IP etc as Cloudflare meaning it can be difficult to get the data to analyse.
Also you need to be on an enterprise plan to get the raw data – which can be too costly for small sites, who want the benefits of Cloudflare but also be able to analyse their server logs.
That all changed recently with Logflare. Logflare allows you to set up a Cloudflare Worker and all the log requests are sent to Logflare.
What’s even cooler is that you can set up rules so you can have Googlebot going to one folder so it’s easily to analyse and report.
More details and info can be found on Reddit.
I haven’t personally used this as I don’t need to – but the idea is pretty cool and if I had a client using Cloudflare, I will be recommending them to set this up.
It’s about 5 minutes off work, but once set up will allow you to collect a lot of valuable info.
