Ospry Suite

· 5 min read

Nineteen trade websites told the AI crawlers not to read them

We checked the robots.txt of 259 UK trade websites. Nineteen of them name GPTBot, ClaudeBot, CCBot and Google-Extended and tell each one to stay out. Almost none of those owners decided that.

A robots.txt is a plain text file at the root of a website. It is the first thing a crawler reads, and it says which parts of the site that crawler may look at. Most people who own a website have never opened theirs, which is reasonable — until something writes an instruction into it on their behalf.

What one of them actually says

Here is a real example, live at the time of writing, from a timber and fencing supplier: 

User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Amazonbot
Disallow: /

That is OpenAI, Anthropic, Common Crawl, Google’s AI training crawler, Apple, ByteDance, Meta and Amazon, each turned away by name.

Nobody sat down and typed that

Look at the same file across all nineteen sites and the blocked list is near enough identical every time. That is the giveaway. A human making a considered decision about AI would not produce the same eight names in the same order on nineteen unrelated websites.

What produces it is a switch. Cloudflare has a one-click “block AI scrapers and crawlers” setting, and several popular plugins ship something similar. Somebody ticked a box during setup, or a host ticked it as a default, and the file was written accordingly.

Why it matters more this year than last

BrightLocal’s 2026 consumer survey puts 45% of people using AI tools to find local businesses, up from 6% the year before. That makes it the third biggest local discovery channel, behind Google and Facebook.

We should be straight about the other half of that, because plenty of people selling “AI SEO” will not be: Google and the map pack still bring in the overwhelming majority of actual enquiries. This is not where your jobs come from this month. It is where a growing slice of them will come from, and the businesses being recommended now are building the same head start that early Google Business Profile adopters got.

Check yours in two minutes

  1. Type your own address followed by /robots.txt — for example yourbusiness.co.uk/robots.txt.
  2. Read what comes up. It is usually short.
  3. Look for GPTBot, ClaudeBot, CCBot or Google-Extended followed by Disallow: /.

If they are there and you want them there, that is a perfectly legitimate position — it is your content, and some businesses have good reasons to keep it out of training data. The point is that it should be your decision rather than one made for you by a checkbox.

One thing this is not

While we were checking, twelve sites in the same set refused our crawler with a 403. It would have been easy, and wrong, to report those twelve as invisible to AI.

We went back and asked each of them again as Googlebot, then as GPTBot, then as ClaudeBot. All three were let straight in. The only crawler being turned away was ours, because bot protection allowlists the crawlers it recognises and refuses the ones it does not. That is a fact about our user agent, not about their website, and it is a very different thing from a robots.txt that names GPTBot and says no.

We mention it because the difference is easy to miss and we nearly missed it ourselves. A number is only worth publishing once you have tried to disprove it.

What we measured

259 UK trade websites — roofing, fencing, landscaping, driveways, building. Every site we happened to audit, not a filtered sample. The robots.txt of each was fetched and read for named AI user agents. 19 of 259 block at least one; the median blocked list is six.