AI search

Is your website invisible to ChatGPT?

Plenty of trade websites serve thousands of words to a customer and an error message to every AI assistant that asks. The owner almost never finds out, because their own browser always shows them the good version. We measured how often it happens: we checked 259 of them.

What actually happens when someone asks an assistant

When a customer types “who’s a good fencer near Leicester” into ChatGPT or Gemini, the assistant doesn’t know the answer from memory. It runs searches, fetches a handful of pages, reads them, and writes a reply from what it found.

That fetching step is the bit that matters. The assistant requests your page using its own named crawler — not Chrome, not Safari. If your hosting, firewall or security plugin refuses anything that isn’t a normal browser, the assistant gets nothing. You are simply not in the answer.

One landscaping website we checked served 3,015 words to a visitor and 15 words with a 403 error to every assistant. Every page. For months.

Answer-time bots and training bots are not the same thing

This is where most advice goes wrong, and where a lot of site owners have blocked the wrong thing on purpose.

Training crawlers — GPTBot, ClaudeBot, Google-Extended, CCBot — collect content to train future models. Blocking them is a legitimate preference. It has no bearing on whether you appear in answers today.

Answer-time crawlers — OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot — fetch your page while a customer is waiting for a reply. Block these and you cannot be recommended, because the assistant never sees you.

Plenty of businesses pasted a “block AI bots” snippet into their robots.txt in good faith and quietly removed themselves from being found. It is the single most common own goal we see.

The Googlebot trap

Worth knowing separately: Google’s AI Overviews — the summary at the top of the results — are served using Googlebot, not Google-Extended. So blocking Google-Extended keeps you out of Gemini training while leaving normal search and AI Overviews untouched. Blocking Googlebot takes you out of everything at once.

Three reasons an assistant sees nothing

1. The door is shut. Your server returns 403 or 401 to anything that isn’t a browser. Usually a firewall rule or a security plugin, and fixable in minutes once you know.
2. The content isn’t there yet. The page returns 200, but the words only appear after JavaScript runs. Assistants generally don’t wait for that. Needs server-side rendering.
3. Nothing is wrong. Genuinely common, and worth confirming rather than assuming.

How to check your own site

Ospry Suite fetches your page twice — once the way an assistant does, once the way a browser does — then compares the word counts and names the cause in plain English. It runs on every audit, not as an upsell.

You can also check roughly by hand: open your robots.txt (yoursite.co.uk/robots.txt) and look for any rule blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot. That catches the deliberate blocks, though not the firewall ones, which are the more common problem.

What we won’t tell you

We won’t tell you that you rank third in ChatGPT. Nobody can read a ranking out of an assistant — the output isn’t deterministic, and there is no chart to look at. Any tool quoting you a position in an AI model is inventing it.

What can be measured is whether you’re reachable, and whether you appear among the sources that currently answer the questions your customers ask. That’s what we report, and we say how confident we are each time.