How to see AI bots in your server logs

What are the key facts about AI bots in your logs?

What kinds of AI bots visit my website?

AI companies now run separate bots for separate jobs. Knowing which job a bot does tells you what its visit means.

• Training crawlers read pages that may be used to train future AI models. Examples: OpenAI's GPTBot, which "crawls content for training foundation models," and Anthropic's ClaudeBot, which collects content "that could potentially contribute to their training." OpenAI Anthropic
• Search crawlers build the index an AI engine searches when it answers. Examples: OAI-SearchBot for ChatGPT search, Claude-SearchBot, and PerplexityBot, which is "designed to surface and link websites in search results on Perplexity." OpenAI Perplexity
• User-triggered fetchers open one page because one person asked a question right now. Examples: ChatGPT-User, Claude-User and Perplexity-User. Perplexity puts it simply: "When users ask Perplexity a question, it might visit a web page to help provide an accurate answer." Perplexity

The rules differ by kind. OpenAI says of ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply." Perplexity says Perplexity-User "generally ignores robots.txt rules." Anthropic says its bots, Claude-User included, honor robots.txt. So a user fetcher can appear in your logs even on pages you told crawlers to skip.

For which of these to allow or block, see Which AI crawlers should you allow. This guide is about seeing them.

Which bot names should I search my logs for?

Search your logs for these names in the user-agent field. We only list bots whose official pages we read on 6 October 2026. Companies rename and add bots, so check the source page before you write a rule.

A full user-agent is long. OpenAI's search bot, for example, sends a browser-like string that ends compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot. Search for the short name, not the whole string, so version changes don't break your filter.

Why doesn't Google Analytics show AI bots?

Two reasons, and either one alone would hide them.

GA4 only counts visitors who run its code. Google Analytics collects data through the Google tag, a JavaScript snippet you paste into the <head> of each page. Google A visit is only recorded if that script runs. Vercel, a hosting company, studied AI crawler traffic on its network in December 2024 and reported that "none of the major AI crawlers currently render JavaScript," naming GPTBot, ClaudeBot and PerplexityBot. They download the page's text and leave. The exceptions Vercel found were Gemini, which uses Googlebot's infrastructure, and Applebot, which renders JavaScript. Vercel

GA4 throws known bots away on purpose. Google says "Traffic from known bots and spiders is automatically excluded," and "you cannot disable known bot traffic exclusion or see how much known bot traffic was excluded." Google

So GA4 isn't broken. It's built to measure people. The bot visits happen one step earlier, on your server, and that's where they're recorded.

Where do I find my server logs?

Every request to your site passes through a web server or a CDN (a content delivery network, the layer many sites sit behind). That layer writes a line for each request: the time, the page, the status code, the IP address and the user-agent. Where you read it depends on how your site is hosted.

If your site is behind Cloudflare. Cloudflare's AI Crawl Control is "Available on all plans" and needs no setup. Cloudflare Its Overview tab shows total AI crawler requests, status codes, the most requested pages, and crawlers grouped by company. The Crawlers tab lists each crawler with request counts and allow or block controls. Referral data, meaning people arriving from AI services, is on paid plans only. Cloudflare

If your site is on Vercel. The Logs section shows runtime logs, and each request's details include the "Request User Agent." Two limits matter. Runtime logs cover functions and middleware, but for static pages only requests served from cache; Vercel says "to get all static logs check Log Drains." And retention is short: 1 hour on Hobby, 1 day on Pro, up to 30 days with Observability Plus. Log Drains, which export logs elsewhere, are on Pro and Enterprise plans. Vercel Vercel

Other hosts. Most hosts keep raw access logs somewhere in the control panel, or will turn them on if you ask. Ask your host or developer three questions: where are the raw access logs, do they include the user-agent and IP address, and how many days are kept?

How do I know a bot is really who it says it is?

Anyone can type GPTBot into a user-agent. Google's own verification guide exists because site owners worry about "spammers or other troublemakers" claiming to be Google. Google So before you trust a line in your logs, check the IP address it came from.

Method 1: match the IP against the company's published list. These files list the IP ranges each bot uses:

Sources: OpenAI, Anthropic, Perplexity, Google, Bing. Apple publishes an Applebot IP list too, linked from its Applebot page. Apple Google notes the ranges are written in CIDR format, a short way of writing a block of addresses.

Method 2: a two-way DNS check. Google describes this for its crawlers: Google

1. Run a reverse DNS lookup on the IP from your logs, for example host 66.249.66.1.
2. Check that the name ends in googlebot.com, google.com or googleusercontent.com.
3. Run a forward DNS lookup on that name.
4. Check that it returns the same IP you started with.

Apple says Applebot traffic is "generally identified by using reverse DNS in the *.applebot.apple.com domain." Apple

If a request says ClaudeBot but its IP isn't on Anthropic's list, treat it as an imposter, not as Claude. Anthropic also warns that blocking by IP instead of robots.txt "may not work correctly or persistently guarantee an opt-out," since its IPs can change. Anthropic

What should I do with what I find in my logs?

This is our advice. The point of reading the logs is to find problems AI engines hit on your site that you can't see from your own browser.

1. Check that the search crawlers and user fetchers are there at all. If you see no OAI-SearchBot, Claude-SearchBot, PerplexityBot or bingbot in a week of logs, something may be blocking them. Start with your robots.txt and CDN rules.
2. Look at the status codes they got. A 200 means they got the page. A 403 usually means a firewall or bot rule turned them away. A 404 means they asked for a page that no longer exists. A 5xx means your server failed. Fix the 403s and 5xx errors on important pages first.
3. See which pages they read. Cloudflare's "Most Popular Paths" table does this for you. Cloudflare If bots spend their visits on old tag pages and never reach your services or pricing pages, improve your internal links and sitemap.
4. Treat user-triggered fetches as live demand. A ChatGPT-User or Perplexity-User hit on your pricing page means someone just asked an assistant something your pricing page answers. Make sure that page states its facts plainly in the HTML, not only inside scripts. See Can AI read JavaScript websites?
5. Separate real bots from fakes before you act. Verify the IP first. Blocking a fake GPTBot is fine. Blocking the real OAI-SearchBot because a fake one misbehaved is not.
6. Check again after any change. A new firewall rule, CDN setting or site rebuild can block bots without any visible error to you.

What should be on my AI bot log checklist?

How do I know if bot visits turn into AI answers?

Logs tell you an AI engine read your page. They don't tell you whether it then cited or recommended you. For that, ask the engines directly.

Our Prompt Simulator runs one question across 12 AI engines at once and shows which sources each one cites. If a page is crawled often but never cited, the problem is the page's content, not access. For the basics of what these bots are, see the AI crawlers glossary entry.

Sources

1. Bots — OpenAI. Read Oct 6, 2026.
2. Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic. Read Oct 6, 2026.
3. Perplexity Crawlers — Perplexity. Read Oct 6, 2026.
4. Google's common crawlers — Google Search Central. Read Oct 6, 2026.
5. Verifying Googlebot and other Google crawlers — Google Search Central. Read Oct 6, 2026.
6. About Applebot — Apple. Read Oct 6, 2026.
7. Meta Web Crawlers — Meta. Read Oct 6, 2026.
8. bingbot IP ranges (bingbot.json) — Microsoft Bing. Read Oct 6, 2026.
9. Set up Analytics for a website and/or app — Google Analytics Help. Read Oct 6, 2026.
10. Known bot-traffic exclusion — Google Analytics Help. Read Oct 6, 2026.
11. AI Crawl Control — Cloudflare. Read Oct 6, 2026.
12. Analyze AI traffic — Cloudflare. Read Oct 6, 2026.
13. Runtime Logs — Vercel. Read Oct 6, 2026.
14. Logs — Vercel. Read Oct 6, 2026.
15. The rise of the AI crawler — Vercel. Read Oct 6, 2026.