See what each AI crawler gets from your pages
ai-readable is a free command line tool and GitHub Action. It shows the robots.txt verdict for every documented AI crawler, how much of a page exists before JavaScript runs, and nine AI readability checks. In CI it fails a pull request when a deploy makes a page unreadable to AI search.
npx ai-readable example.com/pricing --renderView on GitHub npm package Try it on the demo site
What does ai-readable check?
It checks nine things a crawler can read from the page as served: whether search and assistant bots are allowed in robots.txt, whether the page is indexable, one descriptive H1, question-shaped headings, a direct answer under each heading, entity structured data, tables and lists, and a title with a meta description. With the render flag it also loads the page in headless Chromium and reports how much of the content only exists after JavaScript.
| Check | Points | Passes when |
|---|---|---|
| AI crawler access | 25 | No search-index or assistant-fetch bot is blocked for the path |
| Reachable and indexable | 15 | Status below 400, no noindex in meta robots or X-Robots-Tag |
| One descriptive H1 | 10 | Exactly one H1 of three or more words |
| Question-shaped headings | 12 | At least two H2 or H3 headings phrased as questions |
| Liftable answer blocks | 15 | At least two headings followed by a 25 to 90 word paragraph |
| Entity structured data | 10 | JSON-LD naming an Organization, Product, Article or similar |
| Tables and lists | 8 | At least one table or three list items |
| Title and meta description | 5 | Title of 15+ characters, description of 50+ |
Why does the initial HTML matter?
Most AI retrieval crawlers do not execute JavaScript. A pricing page that renders entirely on the client is a title and an empty div to them, so it cannot be cited no matter how good it looks in a browser. ai-readable compares the HTML a crawler downloads with the page after JavaScript ran and reports the gap, with the headings that only appear after rendering.
How does the CI gate work?
You commit a baseline of your key pages. On every pull request the Action checks the preview deployment against that baseline and fails when a page regresses: a search bot newly blocked, a noindex that leaked from staging, a score drop, or content that moved behind JavaScript. It posts one comment on the pull request with the evidence and a link to the fix recipe, and refreshes a README badge on your main branch.
npx ai-readable init --base-url https://example.com
npx ai-readable ci --update-baselineWhich AI crawlers does it know?
Nineteen documented bots from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon, DuckDuckGo, Mistral and Common Crawl, each with a link to the vendor's own documentation and grouped by what blocking it costs you.
- Search index bots build the index that live answers retrieve from. Blocking them removes your pages from those answers.
- Assistant fetch bots open a page on demand when a user asks about it. Blocking them breaks link reading and citations.
- Training bots collect content for model training. Blocking them does not change what live answers can cite.
What does it not do?
It never spoofs bot user agents, because CDNs verify real crawlers by IP and a spoofed request would tell you about the CDN rather than the crawler. It also does not know whether AI engines mention or cite you. A green check means they can read the page. Tracking whether they use it is what Citlyze does.