What does AI crawler analytics answer?
It answers a question most analytics tools cannot. Are AI engines actually reading my site, and which pages? Google Analytics will not tell you, because bots do not run the tracking code. Your server logs will, because they record every request.
The term AI crawler analytics refers to reading your server logs to see which AI bots visited, how often and which pages they fetched.
The trouble is that logs are long, raw text files. Few marketers want to search them by hand. This feature reads them and pulls out the AI bots for you.
How does it work?
You send the logs and the platform does the sorting.
- Connect your access logs. Send them from your web server or content delivery network, for example Caddy logs or Cloudflare Logpush, or with a small scheduled job. The crawler screen explains each option.
- Keep them flowing. New lines are sent as they are written, so the figures stay current.
- Match the bots. Each request is checked against the user agents of known AI crawlers.
- Count and group. Visits are counted by bot and by day, and the most fetched pages are listed.
- Read the results. See which bots came, how often, when each was last seen and which pages were fetched most.
Which AI crawlers does it recognise?
It recognises the main AI bots and more besides. The table shows some of the ones you are most likely to see and who runs them.
| Bot | Run by | What it is broadly for |
|---|---|---|
| GPTBot | OpenAI | Collecting pages that may be used to train models |
| OAI-SearchBot | OpenAI | Finding pages for ChatGPT search |
| ChatGPT-User | OpenAI | Fetching a page when a user asks ChatGPT to |
| ClaudeBot | Anthropic | Collecting pages for Claude |
| PerplexityBot | Perplexity | Finding pages for Perplexity answers |
| Meta-ExternalAgent | Meta | Collecting pages for Meta's AI work |
| Bytespider | ByteDance | Collecting pages for ByteDance |
| CCBot | Common Crawl | Building an open web archive used by many AI projects |
| Amazonbot | Amazon | Collecting pages for Amazon services |
Our glossary entry on AI crawlers explains the different types in more detail.
What do you see?
You see each AI bot with its number of visits over time and when it was last seen. You also see the pages AI bots fetch most. That quickly shows whether bots reach your important pages or spend their time on old tags, filters and archive pages.
Three patterns come up often. A bot that never appears may be blocked in robots.txt, sometimes by accident. A bot that only fetches the home page may not be finding the rest of the site. And a sudden drop after a site change is often a sign that something in the new setup is turning bots away.
Why does it matter for AI search?
AI engines that search the web can only use pages they can fetch. If OAI-SearchBot or PerplexityBot never reach your service pages, those pages have little chance of being read when a buyer asks a question. Crawler data shows whether that first step is happening.
It is not the whole story. A crawl is not a citation, and a page fetched every day can still be ignored in answers. AI citation tracking shows which pages actually get cited. Crawler data sits one step before that.
It also helps you check your own rules. Many sites changed robots.txt in a hurry when AI crawlers first appeared. Our guide to AI crawlers and robots.txt explains the options. The logs show whether the rules you set are doing what you meant. If you are thinking about an llms.txt file, our llms.txt guide covers what it is and what it is not.
What should you do with what you find?
Most findings lead to one of a few simple actions.
- If a bot you want is missing, check robots.txt and any firewall or bot protection rules.
- If bots skip your key pages, check internal links and your XML sitemap.
- If bots spend time on thin or duplicate pages, tidy those up or keep bots out of them.
- If a bot you do not want is busy, decide whether to block it in robots.txt.
It helps to compare the weeks before and after each change. That way you can see whether the fix worked rather than assume it did.
What are the limits?
A user agent is only a label, and some scrapers pretend to be well-known bots. So treat odd spikes with some caution. The data is also only as complete as the logs you load. If a content delivery network serves most requests, those logs may sit with the network rather than your server.
Who uses it?
Technical SEO leads use it after site migrations and robots.txt changes. B2B and SaaS teams use it to see whether AI bots reach their documentation and product pages. Agencies use it as part of an AI visibility audit for new clients.
Where it fits
Crawler data is the first link in the chain. Page optimisation checks that the pages bots fetch are well built. AI citation tracking shows which pages get cited in answers. AI traffic analytics shows the visitors those answers send.