AI crawler audit checklist Site and host checked: Audit date and owner: Representative URLs: Intended search access: Intended training preference: robots.txt reviewed on each relevant host: CDN and firewall rule outcomes checked: Crawler identity verified using provider guidance: Response status and actual body inspected: Essential text and links available: Indexing and preview controls reviewed: Public PDF and gated-content behavior checked: Finding, responsible owner, and next action: Retest date and evidence: