AI Crawler Analytics
See what AI reads before it answers.
Connect Cloudflare in minutes. reconnAI shows every verified AI and search crawler, the pages they fetch, and whether those pages turn into citations in ChatGPT, Gemini, Perplexity and Google AI.
Schedule a demoAI crawler activity
Verified requests from 24 named AI and search crawlers
Verified AI requests
48,210 +12.4%Unverified bot names
6,904 12.5% of bot trafficPages fetched
1,842Blocked or error responses
2.1% 403 and 5xxRequests by intent
Top crawlers
| Crawler | Intent | Requests | Verified | 200 OK |
|---|---|---|---|---|
| OAI-SearchBot OpenAI | AI search | 9,812 | 99% | 97% |
| Googlebot Google | Search | 9,140 | 99% | 98% |
| GPTBot OpenAI | Training | 8,420 | 71% | 96% |
| ChatGPT-User OpenAI | Assistant | 6,305 | 98% | 95% |
| ClaudeBot Anthropic | Training | 5,160 | 93% | 97% |
| PerplexityBot Perplexity | AI search | 4,288 | 97% | 82% |
| Claude-SearchBot Anthropic | AI search | 2,940 | 99% | 98% |
Illustrative data on the reconnAI demo brand. Verified counts; Cloudflare samples busy periods, so large sites see estimates.
We measure how often AI visits your site
Visibility shows the outcome. Crawlers show the cause.
If an AI search bot cannot reach a page, that page cannot be cited. Crawler data tells you whether the content you want quoted is actually being read.
Blocked by accident
One robots.txt line or firewall rule stops OAI-SearchBot, and you drop out of ChatGPT citations without knowing why.
Fake bots in your numbers
Scanners borrow GPTBot’s name to probe your site, so raw logs overstate how much AI is really reading.
Read but not cited
AI fetches your page, then a competitor’s page wins the answer. Access is fine; the content or authority is not.
Bot types
Three kinds of AI bot. Three different objectives for your GEO strategy.
Treating every AI bot the same leads to bad robots.txt calls. reconnAI splits them by intent so you can decide what to allow, what to watch and what to fix.
Training
Shapes what AI knows about you
Collects content to train future models. It decides how well the next generation of ChatGPT or Claude understands your brand, products and category, even when no live search happens.
The long game. Blocking it protects content but can leave models with an out-of-date or competitor-led view of you.
AI search
Decides whether you are cited today
Builds the index AI answer engines search when someone asks a question. If it cannot reach a page, that page cannot appear as a source in the answer.
The most direct link to citations. Keep it allowed, and watch for errors and blocks on the pages you want quoted.
AI assistant
Shows real demand, right now
Fetches a page live because a person’s question led the AI there. It is the closest thing to an AI impression your site can see.
Tells you which pages AI visits when people are asking AI questions, so you can check they are winning the prompts you track.
Understand how LLMs read your website
Know exactly which bots read your site, and why it matters.
Verified crawler tracking
Every request is checked against Cloudflare’s verified-bot list, so you see real crawlers and impostors side by side, never blended together.
Know each bot’s intent
Training, AI search, AI assistant and search engine, colour-coded on one chart. Search and assistant bots decide citations today; training bots shape what models know later.
Plain-English crawler guide
What each bot is, why it matters for GEO and what blocking it costs, with a clear verdict for 26 crawlers and controls.
What it is: the crawler behind ChatGPT search results and citations.
If you block it: your pages cannot appear as sources in ChatGPT search.
Spot fake bots
See what unverified “bots” went for and whether your site served them anything. Most are scanners hunting for config files, not AI.
| Path requested | Claimed to be | Served |
|---|---|---|
| /.env | GPTBot | 403 blocked |
| /.git/config | ClaudeBot | 403 blocked |
| /wp-login.php | PerplexityBot | 404 not found |
| /backup.zip | GPTBot | 404 not found |
Every page they fetch
A searchable list of every URL AI crawlers took, and which crawlers took it.
Response codes per crawler
Pages served, redirects, blocks and errors, broken out bot by bot.
24 hours to 12 months
Filter by crawler or company. Saved nightly, so history builds past Cloudflare’s 30 days.
Connect crawls to your AI visibility.
On its own, crawler data is a log. Read against reconnAI’s visibility, citation and sentiment tracking for the same brand, it tells you which page to fix and why.
- 1
Crawled vs cited. Every page plotted by how often AI fetches it and how often it is cited.
- 2
Crawled, not cited, as briefs. Pages AI reads but never quotes, ready to rewrite with reconnAI’s content tools.
- 3
Human visits from AI. GA4 and Search Console sessions from AI platforms, with crawl-to-referral ratio.
Check access
Cited from old copies. Make sure search bots can still reach it.
Protect
Your winners. Keep them fresh and never block their bots.
Deprioritise or fix
Decide if it should rank in AI at all, then fix links and access.
Rewrite first
AI reads it, then cites someone else. Content work pays most here.
Crawled by AI search bots
Technical health for AI.
robots.txt audit per crawler
Blocked, allowed or missing, for every named AI bot.
Error and drop alerts
Hear first when a key bot starts getting 403s, fetches fall sharply, or a search crawler is blocked.
Sitemap coverage
The share of your sitemap each AI crawler has actually reached.
How it works
- 01
Connect
Add a read-only Cloudflare Analytics token in Brand Kit. More log sources are on the way.
- 02
Verify
reconnAI separates real crawlers from impostors and classifies each one by intent.
- 03
Analyse
Crawlers, pages, responses and trends, saved nightly so your history keeps building.
- 04
Act
Read crawler data next to visibility, citations and sentiment for the same brand.
Private by design
Built for regulated teams in finance and health.
- Counts only, never individual requests
- No IP addresses, query strings or referrers
- Queries that ask for them are refused in code
- Read-only access you can revoke at any time
For agencies
One login across every client brand.
Switch between client sites, share filtered views, and report crawler activity alongside the AI visibility work you already deliver.
Coming soonExports and scheduled client reports
“Knowing whether your brand shows up in AI answers is only half the picture. You also need to know how ChatGPT, Claude and Perplexity are interacting with your site: which pages they read, which they can’t reach, and which they read but never quote. Put the two together and you stop guessing what to fix.”
Questions about AI crawlers
What is AI crawler analytics?
It tracks which AI bots visit your site, which pages they fetch and what your server sends back. That shows whether ChatGPT, Gemini, Perplexity and Google AI can actually read the content you want them to cite, and where access, errors or blocks are getting in the way.
What is the difference between AI training bots and AI search bots?
Training bots such as GPTBot and ClaudeBot collect content for future models. Search and assistant bots such as OAI-SearchBot, ChatGPT-User and PerplexityBot fetch pages to answer questions right now, so they decide which sources get cited today. reconnAI shows each group separately.
Will blocking GPTBot remove me from ChatGPT?
No. ChatGPT search uses OAI-SearchBot, and live fetches during a conversation use ChatGPT-User. GPTBot only controls training. Blocking Googlebot is different: it removes you from Google search and AI Overviews, so treat search engine crawlers with care.
How does reconnAI tell real crawlers from fake ones?
It uses Cloudflare’s verified-bot check, which confirms a request really came from the bot’s operator. Requests that only borrow a bot’s name are shown separately, along with the paths they tried and what your site served, so fake traffic never inflates your numbers.
What data does reconnAI collect?
Counts only: time, hostname, page path, response status and the verified-bot label. No IP addresses, query strings, referrers or individual requests are stored, and the code refuses any query that asks for them. Access is read-only and you can revoke it at any time.
How do I connect my site?
Create a read-only Cloudflare Analytics token and add it in Brand Kit. Your site needs to be served through Cloudflare with records proxied; DNS-only sites can connect using Cloudflare’s Orange-to-Orange option. More log sources are on the way.
How does crawler data help my AI visibility?
It shows whether AI can reach the pages you want cited and which pages it reads most. Read alongside reconnAI’s citation tracking, it reveals pages that are crawled but still not winning answers, which is where content and authority work pays off most.