Last month I found out one of my client’s sites had been blocking Bingbot for over a month. Not on purpose. A WAF rule update did it quietly, in the background, with no warning to anyone. Rankings dropped steadily the whole time, but nobody noticed until traffic was down enough to ask why.
That’s the problem with bot access, there’s nothing to actually tell you when it breaks.
You can have perfect schema markup, a clean site structure, and content that should be ranking and if GPTBot, ClaudeBot, or Googlebot can’t actually reach the page, none of it matters. The bot never sees it, it can’t index it, can’t cite it, can’t put it in front of anyone.
This isn’t a hypothetical problem, it happens constantly to websites all across the world, and almost always silently:
- A WAF rule gets tightened and starts blocking crawler user-agents along with the bad traffic it was meant to catch
- A robots.txt edit meant to block one bot accidentally blocks several others with similar names
- A migration goes live and nobody checks whether the new server config still allows AI crawlers through
- Wordfence or another security plugin starts rate limiting bots that look “too aggressive,” even though they’re legitimate
You won’t see any of this in Google Search Console. You won’t see it in your analytics.
The bot just stops showing up in your logs, and you have no way of knowing why unless you go and test it yourself, bot by bot, manually setting user-agent headers – which nobody has time to do daily. You could use a manual testing tool instead, like Nik Ranger’s AI & SEO Bot Crawlability Tester which are incredibly helpful for testing when you need to see how accessible your site is to bots. Or you could use a fully automated solution like UAWatch.
So I built UAWatch
UAWatch checks 64 bots against your site every single day, including AI crawlers like GPTBot, ClaudeBot, and Perplexity, search engines like Googlebot and Bingbot, SEO tools like Ahrefs and Semrush, and social platforms like LinkedIn and Twitter. It tests both your robots.txt rules and the actual HTTP response each bot gets.
If anything changes, for example, a bot that was allowed yesterday is now blocked, you get an email. That’s it. No noise, no daily report to wade through. Silence means everything’s fine – No news is good news! An email means something needs fixing or has changed.
When something is blocked, UAWatch tells you why in plain English: blocked in robots.txt, blocked by a WAF, rate limited by Wordfence, or just allowed with no restrictions. You don’t need to be a technical SEO wizard to understand what’s wrong or who to tell about it. There’s even a handy PDF download that you can forward to your web developer.
Why This Matters More Now Than In 2024
Generative Engine Optimisation (GEO) only works if the AI crawlers can get to your content in the first place. You can optimise entities, write perfect JSON-LD, and structure your content exactly the way ChatGPT or Perplexity wants it, but if GPTBot is blocked at the firewall, your business doesn’t exist to that AI system. It’s the same logic as traditional SEO, except almost nobody is checking for it.
I see this gap constantly in SEO audits. Clients assume “the bots can get in” because the site looks fine in a browser. It usually has nothing to do with what a human sees and everything to do with how the server treats a non-browser user-agent. I have another client who was using Umbraco and Cloudflare, unknowingly blocking loads of user agents, including those for Internet Explorer, ChatGPT and Claude without knowing. It’s only when I performed an audit that we found out. I got fed up with checking each user agent manually with Chrome UA Spoofer extension, so I thought I’d create an automated tool to do it for me, and UAWatch was born.
What It Costs
£10 per site, per month and you can cancel any time. No setup, no contract. Add a URL, we run the first check immediately, and from then on it’s daily and automatic.
If you manage client sites, run an agency, or just want to know your own site is actually visible to the systems that matter, then it’s definitely worth the five minutes it takes to set up.
*UAWatch is a trading name of The Structured Data Company.
