The crawlability audit: making sure AI bots and Google can both access your content

Technical SEO crawlability has always mattered — but in 2026 it means something new. The question is no longer just “can Google read this page?” It’s “can Google read it, can Bing’s AI index it, and can the crawlers behind ChatGPT, Perplexity, and Gemini access and understand it?” For most websites, the honest answer is: probably not all of them, and you likely don’t know which ones are blocked. At Choco Media, we run technical SEO crawlability audits as a regular part of client onboarding, and the number of sites that inadvertently block AI crawlers while trying to protect server load has climbed significantly in the past twelve months.

This post is a practical checklist — not a theoretical overview. It’s for marketing managers, in-house SEOs, and developers who want to know exactly what to check, in what order, and what to do when something is broken. We’ll cover the classic crawl foundations (robots.txt, sitemaps, render-blocking JavaScript), then move into the newer AI-specific directives that most audit guides still miss.

By the end, you’ll have a repeatable process you can run on any site, a clear picture of what “fully crawlable” means in 2026, and a short list of fixes that move the needle fastest.

Why Crawlability Audits Look Different Now

Three years ago, a crawlability audit meant: check robots.txt, verify your sitemap, look for 4xx errors, and confirm Googlebot could see your rendered HTML. That work still matters. None of it has gone away. But the ecosystem now includes a meaningful second category of crawler: the large-language-model bots that feed AI products.

GPTBot (OpenAI), Anthropic’s ClaudeBot, Google’s Googlebot-AI (used for AI Overviews), PerplexityBot, and several others now crawl the web continuously. They don’t rank you in the way Google’s classic algorithm does, but they determine whether your content ends up cited in AI-generated answers. For B2B companies especially, AI citation is becoming a primary discovery channel.

• OpenAI’s GPTBot has been active since August 2023 — many sites blocked it reactively and forgot to review that decision.
• Perplexity’s crawler (PerplexityBot) respects robots.txt but also pays attention to structured data quality.
• Google uses separate crawl infrastructure for AI Overviews versus classic search — your robots.txt disallow rules apply to both unless you differentiate by user-agent.
• Anthropic’s ClaudeBot follows robots.txt and asks to be excluded only if you explicitly opt out.

The practical consequence: a blanket Disallow: / under a wildcard user-agent rule, or an overly aggressive crawl-delay, may be shutting out citation opportunities you don’t even know exist.

Step 1 — Audit Your robots.txt

Start here. It’s the first thing any crawler reads and the easiest place to introduce errors that have site-wide impact.

What to look for

• A wildcard User-agent: * block with Disallow: / that was meant to be temporary and never removed.
• Disallow rules that block /wp-admin/ but also accidentally match /wp-admin-tips/ (a real blog slug pattern we’ve seen).
• Missing or incorrect Sitemap: declaration at the bottom of the file.
• AI-bot-specific blocks: search for GPTBot, anthropic-ai, PerplexityBot, Google-Extended, CCBot. If any of these appear under Disallow: /, that crawler is locked out entirely.
• Conflicting rules: a general User-agent: * allow followed by a specific bot disallow — the more specific rule wins, but the logic is easy to misread.

How to check it

Fetch your robots.txt directly: https://yourdomain.com/robots.txt. Read every line. Then run it through Google Search Console’s robots.txt tester (under Settings) to simulate specific user-agents against specific URLs.

A good rule of thumb we follow: any bot you want crawling your content should appear in robots.txt only if you’re explicitly granting or restricting access. If a bot isn’t listed, it inherits the wildcard rules. Know what your wildcard says before you assume everything is fine.

Step 2 — Verify Your XML Sitemap

A sitemap is how crawlers discover pages efficiently without relying entirely on link-following. If your sitemap is stale, malformed, or missing your best content, crawlers will still find most pages — but you’re making their job harder and losing the priority signals a clean sitemap provides.

What a clean sitemap looks like

• Declared in robots.txt with a full absolute URL.
• All URLs return 200 status codes — no 301s, 404s, or canonicalised-away pages in the sitemap.
• No noindex pages included (they shouldn’t be there — they create a contradictory signal).
• Last-modified dates are accurate (many CMS plugins set these to “now” on every crawl, which trains crawlers to distrust them).
• Sitemap index file if you have more than one sitemap (posts, pages, products, images).

Tools that help

• Screaming Frog — crawl the sitemap, cross-reference against discovered URLs, flag mismatches.
• Google Search Console → Sitemaps — submit, monitor indexed count versus submitted count, check for errors.
• Ahrefs Site Audit — surfaces sitemap-specific issues in its technical health report.

A gap between “submitted” and “indexed” is normal but worth investigating. A large gap (say, 400 submitted, 90 indexed) usually points to thin content, crawl budget issues, or duplicate/canonical problems — not just a sitemap error.

Step 3 — Tackle Render-Blocking JavaScript

This is where many modern sites fail quietly. If your page content lives inside a JavaScript framework that renders client-side, Googlebot will crawl a shell — and AI crawlers without a rendering budget will see even less.

Google claims it renders JavaScript, which is true — but it queues rendering separately from crawling. There can be days between when Googlebot first fetches a page and when it processes the rendered version. During that window, the content is effectively invisible. AI crawlers like GPTBot do not render JavaScript at all as of 2026; they read raw HTML responses.

What to check

• Use curl -A "Googlebot" https://yourpage.com and inspect the raw response. Does meaningful text content appear, or just an empty <div id="root"></div>?
• Compare the raw HTML response to the rendered view in a browser. If they differ significantly, you have a rendering dependency.
• Check Google Search Console’s URL Inspection tool — it shows you the rendered HTML Googlebot actually processes and flags JavaScript errors.
• Look for lazy-loaded content (infinite scroll, tab panels, accordion sections) that only appears after user interaction — crawlers don’t click.

Fixes

• Server-side rendering (SSR) or static site generation (SSG) for content-critical pages.
• Dynamic rendering as an interim fix: serve pre-rendered HTML to known bot user-agents.
• Move critical body text out of JavaScript-dependent components for key landing pages and blog posts.

For our clients’ content-heavy pages, we recommend SSR or pre-rendering as a non-negotiable baseline. The AI crawlability benefit alone justifies it — search ranking improvements are a bonus. If you’re exploring how technical SEO fits into a broader content strategy, these rendering decisions are usually the highest-leverage fixes available.

Step 4 — Set AI Crawler Directives Deliberately

This is the newest layer of the audit and the one most guides still skip. You now need a deliberate policy for each major AI crawler, because the default (inherit wildcard rules) may not match your actual intent.

The main AI crawlers and their opt-out mechanisms

• GPTBot (OpenAI) — respects User-agent: GPTBot / Disallow: / in robots.txt. Also respects a meta tag: <meta name="robots" content="noai, noimageai">.
• Google-Extended — controls whether your content trains Google’s generative AI models (separate from search indexing). Block with User-agent: Google-Extended / Disallow: /.
• ClaudeBot (Anthropic) — respects robots.txt. User-agent string: anthropic-ai or ClaudeBot.
• PerplexityBot — respects robots.txt, user-agent: PerplexityBot.
• Diffbot — used by several AI knowledge graph products; user-agent: Diffbot.

The decision framework

Ask three questions for each crawler:

1. Do we want to appear in this AI product’s answers? If yes, allow.
2. Are we comfortable with our content being used for model training? If no, add a specific disallow for that bot.
3. Is server load a concern? If yes, set crawl-delay directives rather than blanket blocks.

For most marketing-led businesses, the answer to question 1 is “yes, absolutely” — AI citation is distribution. The answer to question 2 is nuanced and often a legal/brand decision rather than a technical one. The answer to question 3 is almost never “yes” for well-hosted sites.

Step 5 — Audit Internal Linking and Crawl Depth

Crawlers — both traditional and AI — discover content by following links. If your best pages are buried at crawl depth 4 or deeper, or if they’re orphaned (no internal links pointing to them), they’ll be crawled infrequently and may not be indexed reliably.

• Run a full site crawl with Screaming Frog or Sitebulb and check “crawl depth” per URL.
• Identify orphan pages — pages with zero internal links pointing to them. These are high-risk for de-indexation.
• Check that your highest-value pages (service pages, pillar posts, campaign pages) sit at depth 1 or 2 from the homepage.
• Review your navigation: does it link to all core content areas, or have some sections drifted off-nav over time?

Internal linking also carries topical authority signals. A tightly linked cluster of related posts tells crawlers (and LLMs) that your site has real depth on a subject — which increases the probability of citation. Our AI SEO guide for 2026 covers the topical cluster approach in more detail if you want to go deeper on that angle.

Step 6 — Check Canonical Tags and Redirect Chains

Canonical tags resolve duplicate content — they tell crawlers which version of a page is the “real” one. When they’re misconfigured, crawl budget gets split across duplicate URLs and the wrong page may be indexed.

Common canonical errors

• Self-referencing canonicals that point to a different protocol (http:// instead of https://).
• Paginated pages that all canonicalize to page 1 — blocking crawlers from seeing the full paginated content.
• Canonicals that point through redirect chains (canonical → 301 → 301 → final URL) — only the last URL is treated as authoritative, but the chain wastes crawl budget.
• Missing canonicals on parameterised URLs (sort orders, filters, tracking parameters) that generate near-duplicate content.

Redirect chains

Every hop in a redirect chain costs crawl budget. More than two hops (original URL → final URL) is worth fixing. Map your redirect chains with Screaming Frog’s redirect report and collapse multi-hop chains to single-hop wherever possible.

Step 7 — Structured Data and Schema Markup

Structured data doesn’t affect crawlability directly, but it affects what crawlers do with your content once they’ve read it. For AI products, well-formed schema markup is increasingly how content gets understood and categorised accurately.

• Article / BlogPosting — mark up every post with author, datePublished, dateModified, headline, and description.
• FAQPage — if your content has questions and answers (like this post does via the TL;DR format), FAQPage schema makes those Q&As directly available to AI systems.
• BreadcrumbList — signals site hierarchy to crawlers.
• Organization — on your homepage, this tells AI crawlers who you are, what you do, and how to reference you accurately.

Validate your structured data with Google’s Rich Results Test and Schema.org’s validator. Errors here won’t break crawlability, but they reduce the quality of what AI systems infer from your pages. If you’re building out AI-optimised content from scratch, baking schema in from the start is far easier than retrofitting it later.

Putting It Together: The Audit Checklist

Here’s the condensed version you can run through on any site:

1. Fetch and read robots.txt — check wildcard rules, AI-bot directives, and sitemap declaration.
2. Submit and validate XML sitemap in Google Search Console — check indexed vs. submitted gap.
3. Curl key pages with a Googlebot user-agent — confirm content is in raw HTML.
4. Check URL Inspection in Search Console for rendering errors.
5. Audit AI crawler directives for GPTBot, ClaudeBot, Google-Extended, PerplexityBot.
6. Run full site crawl — export crawl depth, orphan pages, and redirect chains.
7. Review internal linking on top 20 pages by traffic and by business priority.
8. Check canonical tags on duplicate-risk URLs (parameterised, paginated, protocol variants).
9. Validate structured data on key page types.
10. Document what you found and set a re-audit schedule (quarterly is usually right).

Most sites have at least three or four issues across this list. The highest-priority fixes are almost always robots.txt errors (immediate, site-wide impact) and render-blocking JavaScript on key pages (high effort, but the highest leverage for AI crawlability specifically).

If you’d like us to run this audit on your site and put together a prioritised fix plan, get in touch — it’s typically a one-week engagement and the output is a structured report you can hand directly to a developer.