How to audit AI crawler access to your website
A practical guide for small business owners: how to audit AI crawler access and spot rules hiding your best pages from AI, without weakening security.
To audit AI crawler access, check the policy in robots.txt, the rules enforced by your CDN or firewall, the responses recorded in server logs, and the content a visitor can actually retrieve. Separate training crawlers, search crawlers, and user-triggered fetchers before deciding what to allow.
A first pass can be a short checklist exercise. Resolving a problem may require your developer or hosting provider. Passing the checks does not guarantee indexing, citation, or a recommendation.
Step 1: Choose the pages and the policy you want
Start with a small representative set: the homepage, a service or category page, an article, and a page that matters commercially. Add a product page or public PDF if those are important to your business.
Decide which uses you intend to permit. Supporting search discovery is different from permitting content to be used in model training. A blanket allow-or-block decision for everything labeled AI can miss that distinction.
Keep private areas private through authentication and appropriate access controls. robots.txt is a request to compliant crawlers, not a security boundary. Do not expose confidential material as a visibility experiment.
Step 2: Read robots.txt on the correct host
Open your site's /robots.txt file on the actual host that serves the pages. A main domain and a subdomain can have different files. Check the relevant user-agent groups, path rules, and any broader wildcard rules.
Look for unexpected Disallow directives covering the whole site or an important section. Check that the file is publicly retrievable and that a recent migration or security tool has not replaced it.
Do not paste a generic configuration into production without reviewing the existing rules. Your file may intentionally protect crawling resources or express a training preference. Record the current policy and have the responsible owner approve any change.
For a first policy review, use our free AI crawler checker to check robots.txt rules for AI-related agents; this is a policy check, not proof of real bot access. The glossary explains AI user agents.
Which AI user agents should you recognize?
Roles checked against provider documentation on November 11, 2026. Exact versions, verification methods, and policies may change.
| Agent or control | Main role | Audit distinction |
|---|---|---|
| GPTBot | OpenAI crawler for potential model-training use | Separate from ChatGPT search access |
| OAI-SearchBot | OpenAI search crawler | Relevant to ChatGPT search discovery |
| ChatGPT-User | OpenAI user-triggered fetching | Not the control used to determine search inclusion |
| OAI-AdsBot | Checks pages submitted as ChatGPT ad destinations | Separate from organic discovery and training |
| ClaudeBot | Anthropic training crawler | Separate from Claude search and user retrieval |
| Claude-SearchBot | Anthropic search crawler | Check independently from training preferences |
| Claude-User | Anthropic user-requested retrieval | Check the provider's documented fetch behavior |
| PerplexityBot | Perplexity search crawler | Not a foundation-model training crawler |
| Perplexity-User | User-triggered Perplexity retrieval | Provider says these requested fetches generally ignore robots.txt |
| Googlebot | Google's search crawler | Review search indexing and preview controls separately |
| Google-Extended | Policy token for specified Gemini training and grounding uses | Not a separate HTTP crawler identity; not the Google Search inclusion control |
| Applebot-Extended | Policy control for use in Apple foundation-model training | Does not itself crawl pages; distinct from Applebot search crawling |
| CCBot | Common Crawl's public web dataset crawler | Verify identity and evaluate dataset policy separately |
For authoritative details, use OpenAI's crawler documentation, Anthropic's crawler guide, Perplexity's bot documentation, Google's crawler reference, Apple's Applebot guide, and Common Crawl's CCBot page.
Step 3: Check CDN and firewall controls
A permissive robots.txt file does not help if the edge layer rejects or challenges the actual request. Review your host and CDN settings for AI crawler controls, bot challenges, rate limits, and rules inherited from a security preset.
Use security-event records to identify what was blocked and which rule acted. Do not disable protection site-wide to make one test pass. Ask the provider to apply a narrow correction consistent with the intended access policy.
Verify crawler identity using the operator's current published method. A user-agent string can be copied by anyone. A request that says GPTBot is not sufficient proof that it came from OpenAI. Provider verification may involve published IP ranges, DNS checks, or other documented mechanisms.
Step 4: Inspect logs and response content
Ask your host or developer for requests to the selected pages from relevant verified agents. Record the timestamp, URL, status, and rule outcome. Look for repeated forbidden responses, challenges, redirects, or rate limits.
A 200 status alone is not proof that useful content was delivered. The response could contain a challenge screen, an empty application shell, or an error message. Inspect the returned body and confirm that the page's essential information is present.
No observed bot visit does not prove a block. The agent may not have requested that page during the log period, or the logs may be incomplete. Report what you observed and the coverage limit instead of inventing a failure.
Step 5: Check rendering, links, and important files
Make sure the main answer is available in accessible page text. If the site relies heavily on JavaScript, inspect what the relevant retrieval route receives and avoid assuming that every assistant executes the same browser behavior.
Check internal links to the page and look for accidental noindex or restrictive preview settings where search discovery is intended. These are separate from robots.txt and need their own review.
Public PDFs should contain selectable text and useful context. Important gated information may need a public summary that accurately explains what is available, rather than exposing private material. See JavaScript and AI visibility, gated content, and PDF visibility.
Step 6: Record the finding and retest
Write down the intended policy, the observed result, and the evidence. Assign a named owner and a next check. After a change, repeat the same checks and compare the actual response, rather than assuming that saving a setting resolved the issue.
Then assess discoverability separately. Access is a prerequisite for some information routes, not proof that the platform has indexed the page or will use it in an answer. A visibility scan and a fuller AEO visibility audit answer different questions from a server-access test.
Keep access checks in your maintenance routine
A single successful test does not prove continued availability. Vendor changes, new firewall rules, and updated robots files can alter access after the audit. Include crawler checks in release reviews and regular visibility reviews, especially after a migration or a change to bot management.
When citations or recommendations change, investigate access before assuming that content quality is the cause. Keep the policy finding separate from the observed request outcome: a permitted robots.txt rule is not a successful server response, and a missing log entry is not proof of a block.
AI crawler audit checklist
- Site and host checked:
- Audit date and owner:
- Representative URLs:
- Intended search access:
- Intended training preference:
- robots.txt reviewed on each relevant host:
- CDN and firewall rule outcomes checked:
- Crawler identity verified using provider guidance:
- Response status and actual body inspected:
- Essential text and links available:
- Indexing and preview controls reviewed:
- Public PDF and gated-content behavior checked:
- Finding, responsible owner, and next action:
- Retest date and evidence:
Download the AI crawler audit checklist (.txt)
Use this checklist for an initial review. A blank field means the check still needs evidence; it does not mean the site passed. Download it for your developer or hosting provider and keep the completed version with your maintenance records.
Frequently asked questions
How can small business owners audit AI crawler access?
Start with a few important URLs and check robots.txt, hosting or CDN controls, and the actual response content. Ask the host for verified crawler requests and rule outcomes if logs are not available to you. Record what you found and escalate technical changes to the responsible owner rather than disabling security broadly.
How do I make sure AI crawlers can access my website?
Define which uses you want to permit, then verify that the relevant agents can retrieve the intended public content without a block or challenge. Check both policy files and enforced infrastructure rules. Access alone does not ensure indexing or inclusion in answers, so test discoverability separately after resolving any confirmed access problems.
Should I block GPTBot?
That is a policy decision about potential training use, not a shortcut to controlling every ChatGPT experience. OpenAI documents GPTBot separately from OAI-SearchBot and user-triggered retrieval. Review the business's content policy and official documentation before changing rules, and avoid assuming that one bot controls training, search, and all user requests.
Does blocking Google-Extended remove me from Google AI Overviews?
Google-Extended is not the control for inclusion in Google Search. It governs specified Gemini training and grounding uses. Review Google's current Search AI documentation and the separate Search Console generative AI control for applicable search settings. Do not substitute a Google-Extended rule for a deliberate review of search indexing and preview policies.
Is an llms.txt file enough to fix crawler access?
No. A text guidance file does not override a firewall block, authentication requirement, robots policy, or inaccessible page content. Google says no special AI text file is required for its AI search features. Review the actual access path and use optional files only for a clear supported purpose, without promising rankings or citations.
Turn the audit into a useful next step
Run a free AI visibility scan for a quick preview of how AI may describe your brand. For technical and content investigation, explore the AEO visibility audit. Our llms.txt generator is an optional utility, not a replacement for access checks.
