How CrawlReadable scores a site
CrawlReadable makes the same kind of request most AI crawlers make: a plain HTTP fetch of your homepage with no JavaScript, plus robots.txt, llms.txt, sitemap.xml and /openapi.json. It runs 20 pass-or-fail checks worth 100 points in total.
Every check
| Check | Category | Points | Passes when |
|---|---|---|---|
| Publish a robots.txt file | access | 4 | /robots.txt returns a text file, not an HTML page |
| Let GPTBot crawl the site | access | 6 | robots.txt doesn't disallow / for GPTBot (or for * when GPTBot has no group) |
| Let ClaudeBot crawl the site | access | 6 | robots.txt doesn't disallow / for ClaudeBot |
| Let PerplexityBot crawl the site | access | 6 | robots.txt doesn't disallow / for PerplexityBot |
| Add an llms.txt file | access | 3 | /llms.txt is a text file over 30 characters |
| Mirror llms.txt at /.well-known/llms.txt | access | 1 | /.well-known/llms.txt is a text file over 30 characters |
| Put your words in the HTML | content | 18 | the HTML has at least 400 characters of visible text |
| Stop serving an empty app shell | content | 10 | the page isn't a shell: under 300 characters of text with 2 or more scripts |
| Add real copy to the page | content | 4 | visible text is 8% to 90% of the HTML and at least 100 characters |
| Give the page an h1 heading | content | 4 | there's at least one h1 |
| Use semantic HTML tags | content | 4 | at least 2 of article, main, section, nav, header, footer |
| Add structured data (JSON-LD) | structure | 6 | at least one valid JSON-LD object with a @type |
| Describe your organisation in JSON-LD | structure | 4 | JSON-LD includes Organization, WebSite, LocalBusiness or Product |
| Write a descriptive page title | structure | 3 | the title is 10 to 100 characters |
| Add a meta description | structure | 3 | there's a meta description |
| Add Open Graph tags | structure | 2 | at least one og: meta tag |
| Add a canonical link | structure | 2 | there's a canonical link |
| Publish a sitemap.xml | discovery | 6 | /sitemap.xml is a urlset or sitemap index |
| Keep sitemap dates fresh | discovery | 4 | the newest sitemap lastmod is within 90 days |
| Offer a machine-readable interface | interface | 4 | /openapi.json is a valid OpenAPI document, or /llms-full.txt is a text file |
Grades
A+ is 90 and up, A 80 to 89, B 65 to 79, C 50 to 64, D 35 to 49, F under 35.
When there's no score
If your server answers with an error (HTTP 400 or above) or a bot challenge page, we give no score instead of a low one. A block on our fetcher says nothing reliable about how GPTBot or ClaudeBot are treated.
What we can't see
Our fetcher identifies itself as CrawlReadableBot. Firewall and CDN rules that treat named AI crawlers differently, or block them by IP address, are invisible to us. We read your robots.txt rules but can't confirm your server enforces them. We check the homepage only.
Why these weights
Readable content is 40 points because it's the most common failure: a JavaScript-built page that looks complete in a browser and empty to a crawler. llms.txt is 4 points because 2026 log studies found 97% of llms.txt files get no AI requests.
The scoring engine is open: the same rules run in our Python command-line tool and on this site, and produce identical scores.