Short answer: you can see which AI crawlers visit your website by checking three places: your hosting or CDN request logs (or Cloudflare’s analytics), a filtered view of your server access logs, and a simple check of your robots.txt file. Look for user agents such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Googlebot. The whole check takes about thirty minutes, and it tells you whether the AI systems you care about can reach your content at all.
Most advice about AI visibility starts with content. Before content matters, access has to work. If a crawler is blocked by a firewall rule, a robots.txt line or a bot-protection setting, nothing you publish will reach that system through live retrieval. This guide shows how to check, in plain steps, without developer tools you do not already have.
Why look at AI bot traffic at all?
Three reasons. First, to confirm that the bots you want are reaching your site. Second, to find accidental blocks, which are more common than most owners expect, because security plugins and CDNs often block unfamiliar bots by default. Third, to understand the difference between bots that train models, bots that build a search index, and bots that fetch a page when a user asks a question. Each type matters for a different reason.
If you have not yet decided whether to allow or block AI crawlers as a matter of policy, start with our guide to the GPTBot, ClaudeBot and Cloudflare decision, and return here to verify what is actually happening.
Which bots should you know by name?
Names and purposes change, so check each company’s current documentation, but the main ones as of this writing are:
- GPTBot (OpenAI): crawls for model training purposes.
- OAI-SearchBot (OpenAI): builds the index used by ChatGPT search results.
- ChatGPT-User (OpenAI): fetches pages when a user asks ChatGPT to visit or look something up.
- ClaudeBot and related Anthropic agents: crawling for Anthropic’s models and features.
- PerplexityBot and Perplexity-User: indexing and user-triggered fetches for Perplexity.
- Googlebot: the main Google crawler, which also supplies content to Google’s AI features in search.
- Bingbot: powers Bing and feeds Microsoft’s Copilot experiences.
Note that Google-Extended is a robots.txt control token for use in Google’s AI models, not a separate crawler you will see in logs. It matters in robots.txt but will not appear as a visitor.
The difference between training bots and search or user bots is important. Blocking a training bot is a policy choice about how your content is used in model development. Blocking a search or user bot can mean your pages are not available when someone asks a question that you could answer.
Step 1: Check your robots.txt
Open yourdomain.com/robots.txt in a browser. Read it from top to bottom. You are looking for:
- A blanket
Disallow: /underUser-agent: *. That blocks everything, including search engines, and is often left over from a staging site. - Specific lines for AI bots, such as
User-agent: GPTBotfollowed byDisallow: /. - Rules added by plugins or hosting panels that you did not write yourself.
Write down which bots are allowed, which are blocked and which are not mentioned. Not mentioned usually means allowed by default under the wildcard rule.
Step 2: Look at your CDN or hosting analytics
If you use Cloudflare, Fastly, Akamai or a similar service, the dashboard often has a bot or traffic analytics view. In Cloudflare, check the analytics and security sections for traffic by user agent or “verified bot” categories, and look at the firewall events for blocked requests. Many CDNs now offer an AI-crawler control that can block or allow groups of bots with one switch, and sometimes it is turned on by default for new accounts. Open that setting and read it. A single toggle can undo months of publishing.
If you host on a managed WordPress platform, look for a request log or visitor statistics page. If it lists user agents, filter by the names above.
Step 3: Filter the raw access logs
If you have access to server logs, either through a hosting panel or SSH, you can search them. On a typical Linux server with an Nginx or Apache access log, a command like the following counts requests from a given bot:
grep -i "GPTBot" access.log | wc -l
grep -i "OAI-SearchBot" access.log | wc -l
grep -i "PerplexityBot" access.log | wc -l
To see which URLs a bot requested, and what response codes it received, use:
grep -i "ClaudeBot" access.log | awk '{print $7, $9}' | sort | uniq -c | sort -rn | head -20
Field positions vary with the log format, so check a sample line first. What you are looking for is the response code. A 200 means the page was served. A 403 or a 429 means the bot was refused or rate limited. A 404 can mean the crawler is following outdated links. A 301 chain can waste crawl effort.
If you do not have shell access, export the log file from your hosting panel and open it in a spreadsheet tool, then filter the user agent column.
Step 4: Verify that a bot is who it says it is
User agent strings can be faked. If a request claims to be GPTBot but comes from an unrelated address, it may be a scraper. The major AI companies publish IP ranges or verification methods in their documentation, and Google and Bing document how to verify their crawlers through reverse DNS lookups. If you want to be sure, compare the IP addresses in your logs with the published lists. For most small business checks, this level of detail is optional, but it is worth doing before you make blocking decisions based on log numbers.
How do you read the results?
| What you see | What it likely means | What to do |
|---|---|---|
| No visits from search bots at all | Blocked, or the site is new or small | Check robots.txt and CDN settings; confirm a sitemap is submitted |
| Many 403 responses to AI bots | Firewall or bot protection is blocking them | Decide by policy; whitelist the bots you want |
| Visits to only the homepage | Weak internal linking or a thin sitemap | Improve navigation and link to key pages |
| Heavy training-bot traffic, little user-bot traffic | Normal; training crawlers are more frequent | Consider policy, but do not read much into the gap |
| Frequent hits on old or broken URLs | Outdated links or redirects | Fix with redirects and update sitemaps |
What if no AI bots visit?
For a small or new site, low crawler traffic is normal and not necessarily a fault. Crawlers prioritise sites they have already seen linked and cited. To improve your chances, make sure search engines index you (Google Search Console and Bing Webmaster Tools are both worth setting up; our guide to Bing Webmaster Tools and IndexNow shows the steps), publish a clean sitemap and link to your pages from within your own site. After that, run a few prompts yourself. The monthly AI visibility check shows how to track results over time.
Common mistakes
Assuming an SEO plugin handles it. Plugins vary and update their defaults. Check the output of robots.txt rather than trusting the settings screen.
Blocking everything because of a headline. Decide on a policy per bot type. Training and retrieval are not the same.
Forgetting staging rules. A site launched from a staging copy with a global disallow can stay invisible for months.
Over-reading small numbers. A handful of visits a week from a given bot is normal for a small site. Look at trends, not single days.
Thirty minutes of checking will tell you whether the front door is open. Once it is, content and credibility do the rest.