Reading Your Server Logs: Which AI Crawlers Are Actually Visiting Your Store

Reading Your Server Logs: Which AI Crawlers Are Actually Visiting Your Store

Everyone in this field talks about AI visibility as though it starts with what a model says about you. It doesn’t. It starts several steps earlier, with a much duller question: **has anything actually come and read your pages?**

You can answer that. Not with a tool, not with a model, not with an estimate — with your own server logs, which contain a complete, unfalsifiable record of every request made to your store. It is the only ground-truth data source in this entire discipline, and almost nobody looks at it.

I spent a fair bit of time building crawler tracking into GEOptimisation, and the reason is that when I started checking logs on real stores I kept finding the same three things: crawlers arriving far more often than merchants assumed, crawlers being accidentally blocked, and crawlers fetching pages that turned out to be nearly empty when rendered. All three are invisible from the outside and all three are fixable in an afternoon.

The bots, and what each one is actually for

First, precision, because this is where most advice goes wrong. “AI crawler” is not one thing. The major providers run several bots with genuinely different jobs, and blocking the wrong one has very different consequences from blocking the right one.

From OpenAI’s own published bot documentation:

– **`GPTBot`** — crawls content that may be used to train foundation models.
– **`OAI-SearchBot`** — surfaces websites in ChatGPT’s search features. **This is the one that gets you cited.**
– **`ChatGPT-User`** — user-initiated fetches, when someone asks ChatGPT to look at a page.
– **`OAI-AdsBot`** — validates the safety of pages submitted as ads; OpenAI states its data is not used for training.

Anthropic runs a comparable split — a training crawler, a search indexer and a user-triggered fetcher. Perplexity runs `PerplexityBot` for periodic indexing and a separate user-triggered agent for live retrieval. Google uses standard **`Googlebot`** for AI Overviews and AI Mode, with **`Google-Extended`** existing purely as a training opt-out token rather than a crawler in its own right — a distinction that trips up a lot of people. Apple’s `Applebot-Extended` works the same way. Amazon, Meta and Common Crawl each run their own.

The critical distinction, and I’d underline it: **training crawlers and search crawlers are different bots with different consequences.** Blocking `GPTBot` keeps your content out of future model training and costs you nothing in citations. Blocking `OAI-SearchBot` removes you from ChatGPT’s search results. Those are wildly different decisions, and I have seen stores make the second one by accident while intending the first.

Verify current tokens against each provider’s own documentation before you act on any of this — they change, and third-party lists go stale fast.

Three things I keep finding in logs

**One: accidental blocks.** Usually inherited. A robots.txt line copied from a blog post in 2023 when “block the AI scrapers” was the prevailing mood, a security app’s default rules, a CDN bot-management setting nobody configured, or a firewall rule that rate-limits anything non-browser-shaped. The merchant has no idea. The logs show a crawler arriving, getting a 403 or a 429, and not coming back.

**Two: crawl concentration.** Bots fetch the homepage and a handful of collection pages, and barely touch the product catalogue. If a crawler has seen forty of your four thousand products, then whatever a model says about your range is an extrapolation from 1% of it.

**Three: successful fetches of empty pages.** A 200 response is not the same as a useful page. If your product content — specs, reviews, key attributes — is injected client-side by an app after page load, the crawler may be receiving a shell. Status code says success. Content says nothing.

That third one is the most expensive and the least visible, because every dashboard you own will tell you everything is fine.

Getting at the data

On Shopify you don’t get raw logs from the platform, which is the standard objection to this whole exercise. There are three routes around it.

**A CDN or proxy in front of your store.** If you run Cloudflare or similar, its analytics will break traffic down by user agent, and you can filter to bot traffic directly. This is the cleanest route and many stores already have it.

**A logging app or server-side tag** that records user agent, path, status code and timestamp for each request. More work, more control.

**A tool that does it for you** — which, transparently, is part of what I built GEOptimisation to do, because doing it manually on a client store the first few times convinced me nobody was going to keep doing it manually.

Whichever route, the four fields you need are the same: **user agent, path, status code, timestamp.** Everything useful comes out of those four.

The five questions to ask of the data

Once you have thirty days of it:

**1. Which bots are visiting, and at what frequency?** Build the list. Note who is absent. Absence is a finding — if `OAI-SearchBot` has never appeared, you are not in ChatGPT’s search index and no amount of content work will change that until you fix why.

**2. What status codes are they receiving?** Anything other than 200 on a page you want cited is a problem. Cluster the 403s, 404s, 429s and 5xxs and look at what they have in common.

**3. What proportion of your catalogue has been crawled?** Count distinct product URLs fetched against total product URLs. On most stores this number is embarrassing, and it’s the number that best predicts whether a model can speak accurately about your range.

**4. What are they crawling most?** If bots are spending their budget on filtered collection URLs, tag pages and paginated duplicates, that’s crawl budget you’re not spending on products. Faceted navigation is usually the culprit.

**5. Is crawl frequency trending up or down?** More useful than any absolute number. A declining trend on a store that hasn’t changed usually means something is being blocked or slowed.

What to do with what you find

Fix blocks first — it’s the cheapest fix in the entire field and it’s binary. Audit robots.txt line by line, check your CDN’s bot rules, check any security app’s defaults, and confirm what each rule actually does rather than what you assume it does. Decide deliberately on training crawlers; be extremely careful with search crawlers.

Then fix render. Take three product pages a crawler has fetched and check what’s in the raw HTML rather than the rendered page. If your specs and reviews aren’t there, that’s your priority, not your content calendar.

Then fix concentration. A clean sitemap, sane internal linking and fewer junk URLs competing for attention will spread crawl across the catalogue.

Why this is the first thing, not the fifth

There’s a hierarchy in this work that took me too long to see. Being crawled is a precondition for being indexed, which is a precondition for being cited, which is a precondition for being recommended. Every one of those depends on the one before it.

Which means optimising your product descriptions while `OAI-SearchBot` is getting a 403 at your firewall is like rewriting a shop window nobody can reach the street to see.

Start at the bottom. The logs will tell you, in a few hours, whether the rest of your effort has any chance of mattering — and unlike almost everything else in AI search, they’ll tell you the truth rather than an estimate.

*Sources: OpenAI bot documentation (developers.openai.com); published crawler references for Anthropic, Google, Perplexity, Apple, Amazon, Meta and Common Crawl. Verify all user-agent tokens against each provider’s own documentation, which changes.*