Short answer
To make your website readable by AI, do four things. First, publish an llms.txt file — a plain-text map of your key content proposed by Jeremy Howard in September 2024 — as a low-cost bet, since adoption is real but contested. Second, audit robots.txt so you are not accidentally blocking GPTBot, ClaudeBot, or PerplexityBot from the answers you want to appear in. Third, mark up pages with the few schema.org types that reliably surface (Article, FAQPage, Organization, Product). Fourth, measure exposure separately from clicks: track AI citations and fetch logs, because being cited rarely sends traffic. None of these is a guaranteed ranking lever, but together they remove the friction that keeps AI from reading and quoting you.
What is llms.txt, and does anyone actually use it?
llms.txt is a plain-text file you place at your domain root to hand AI models a curated map of your most important content in clean Markdown — adoption is real and growing, but genuinely contested. The format was proposed by Jeremy Howard in September 2024 (llmstxt.org) so language models could read structured text instead of parsing cluttered HTML.
How common is it? The numbers diverge sharply with the sample. As of June 2026, 8.7% of the world's top 1,000 websites publish an llms.txt file, per Rankability's adoption tracker published July 8, 2026 (Rankability). An SE Ranking study of 300,000 domains, reported by limy.ai on May 12, 2026, put adoption at 10.13% (limy.ai). Casey Burridge's analysis on June 20, 2026 found 1.04% of the top-10k domains had a valid llms.txt in July 2025, rising to 5.61% by June 2026 — roughly 5.4x growth in twelve months (caseyrb.com).
The honest caveat: Digital Strategy Force reported on April 20, 2026 that adoption sat at just 2.13% in 2025, that 39.6% of llms.txt files are empty plugin stubs, and that no major AI crawler has officially committed to consuming the format (Digital Strategy Force). Treat llms.txt as a cheap early-mover bet, not a proven ranking lever.
robots.txt vs AI crawlers: what blocking really costs you
Your robots.txt file decides whether AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot can fetch your pages — and blocking them removes you from the exact answers you want to be cited in.
Publishers are blocking aggressively. BuzzStream reported on April 8, 2026 that 79% of top news sites block AI training bots via robots.txt, 71% also block AI retrieval bots, and PerplexityBot is blocked by 67% of them (BuzzStream). Across a wider sample, ai-visibility.org.uk measured 1,744 sites on March 28, 2026 and found 9.9% block GPTBot in robots.txt (ai-visibility.org.uk).
The crawl economics explain the tension. A Cloudflare network analysis for Q1 2026, reported by technologychecker.io on April 3, 2026, found GPTBot is the most blocked AI crawler, and that ClaudeBot crawls 20,583 pages for every single referral it returns (technologychecker.io). That imbalance is why so many sites block — but the fix is precision, not a blanket ban.
- Training bots (like GPTBot) feed future models; blocking them limits how models learn about you long term.
- Retrieval bots fetch pages live to answer a query; blocking these removes you from real-time AI citations today.
- Decide per bot, not per company — allow retrieval you benefit from, restrict training you don't.
If you are unsure what your file currently allows, an AI visibility audit will surface every unintended block.
Which schema.org types actually surface in AI answers
Structured data helps AI systems understand what a page is about, but only a small set of schema.org types reliably feeds AI answers — and much of the markup sites ship does nothing.
The types that consistently help models identify and quote your content are Article and NewsArticle (author, publish date, headline), FAQPage (question-and-answer pairs models can lift verbatim), Organization (entity identity and sameAs links), Product (name, price, availability), and Breadcrumb for site structure. HowTo markup helps for procedural queries. These work because they mirror how AI answers are assembled: a claim, a source, and a clear entity behind it.
What does not work is just as important. Schema that contradicts the visible page, review markup on pages with no real reviews, keyword-stuffed descriptions, and orphaned JSON-LD blocks unrelated to page content are ignored or, worse, erode trust. In our experience running these engagements at LYVIA, the biggest wins come from making schema honest and complete rather than adding exotic types — accuracy and match to on-page content matter more than volume.
What is actually measurable today
You can measure three things today: AI citations (were you referenced), fetch logs (did AI bots retrieve you), and AI-attributed traffic (did anyone click) — and you must keep exposure and clicks in separate columns, because they rarely move together.
Freshness is a measurable edge. An Ahrefs study of roughly 17 million citations, reported by Crowdo in 2026, found AI-assistant citations were 25.7% fresher by publication age than organic results overall (Crowdo). But exposure seldom converts to visits. A Search Engine Journal field experiment in 2026 found Google AI Overviews reduced organic clicks by 38% on triggered queries (Search Engine Journal).
Two numbers make the click gap concrete: Pew data from March 2025, cited by webiano.digital, found clicks on sources inside Google AI summaries occurred in only 1% of visits with an AI summary (webiano.digital), and Searchless.ai reported on March 24, 2026 that 93% of Google AI Mode searches end without a click (Searchless.ai).
The takeaway: track citations and fetch logs as your primary AI KPIs, and read the broader click impact in our AI Overviews traffic impact analysis.
How to make your website readable by AI, step by step
Making your website readable by AI is a short technical checklist you can complete in an afternoon, then maintain. Work through it in order.
- Audit robots.txt first. Confirm you allow the retrieval bots you want (GPTBot, ClaudeBot, PerplexityBot) and only restrict training where you have a reason to.
- Publish a real llms.txt. List your key pages with clean Markdown links and one-line descriptions — not an empty plugin stub.
- Mark up pages with the schema that surfaces: Article, FAQPage, Organization, Product. Make it match the visible page exactly.
- Keep content fresh and factual. AI engines favor recency, so dated, well-sourced pages get cited more often.
- Instrument measurement. Watch server fetch logs for AI user agents and track where you are cited, separate from traffic.
This is the foundation layer of a broader generative engine optimization program. LYVIA is a Paris-based AI agency, and we run this exact checklist for US and UK clients who want to be readable before their competitors are.
Common mistakes that keep AI from reading your site
Most sites that fail to appear in AI answers are not penalized — they are simply unreadable or accidentally blocked. These are the recurring causes.
- Blanket-blocking every AI bot in robots.txt, which also removes you from live retrieval and citation.
- Shipping an empty llms.txt stub from a plugin, which signals nothing — the same problem Digital Strategy Force flagged in 39.6% of files on April 20, 2026.
- Schema that contradicts the page, which gets ignored and erodes entity trust.
- Content behind heavy JavaScript that retrieval bots render poorly or not at all.
- Measuring only clicks and concluding AI "doesn't work," when the real metric is citation and exposure.
Fix these before adding anything exotic; the fundamentals return most of the value.
llms.txt vs robots.txt vs schema: which matters most
If you can only fix one thing, fix robots.txt — it is the gatekeeper that determines whether AI can read you at all. Schema comes next because it shapes how you are understood, and llms.txt is a promising but unproven bet on top.
Think of them as a hierarchy of certainty. robots.txt has a definite, immediate effect: block a retrieval bot and you disappear from its answers today. Schema.org has a strong, well-established effect on machine comprehension. llms.txt has an uncertain effect — worth publishing because it is cheap and low-risk, but with no major crawler officially committing to it as of April 2026. Sequence your effort by that certainty, not by novelty.
Getting the order right is what separates sites that get cited from those that get crawled and ignored — the difference we detail in how to get cited by ChatGPT.
Cost and effort: what this actually takes
The technical work to make a small-business website readable by AI is low-cost: the files are free, the changes are small, and most of the effort is decisions rather than development. What costs more is doing it well and maintaining it.
A robots.txt audit and a hand-written llms.txt are typically an afternoon of work. Schema markup scales with page count but is straightforward for a standard site. The recurring cost is measurement and content freshness — keeping pages current and watching fetch logs and citations over time. In our experience running these engagements at LYVIA, the setup is quick; the ongoing discipline of measuring exposure separately from clicks is where teams need the most support. For a structured combined approach, see our SEO and GEO content strategy.
Frequently asked questions
What is llms.txt and do I need one?
llms.txt is a plain-text file at your domain root that gives AI models a curated Markdown map of your key pages. It was proposed by Jeremy Howard in September 2024. You should publish one as a low-cost early-mover bet, but not expect guaranteed results: adoption estimates for 2026 range from 8.7% of the top 1,000 sites (Rankability, July 8, 2026) to 10.13% of 300,000 domains (SE Ranking via limy.ai, May 12, 2026), and no major AI crawler has officially committed to consuming the format as of April 2026.
Does blocking AI crawlers in robots.txt hurt my visibility?
Yes. Blocking retrieval bots like GPTBot, ClaudeBot, or PerplexityBot in robots.txt removes you from the live AI answers those systems generate. Many publishers block aggressively — BuzzStream reported on April 8, 2026 that 79% of top news sites block AI training bots and 67% block PerplexityBot — but blanket blocking also cuts off citations you want. The better approach is to decide per bot: allow the retrieval that benefits you and restrict only the training you have a reason to withhold.
Which schema.org types actually help with AI answers?
A small set does the heavy lifting: Article and NewsArticle, FAQPage, Organization, Product, and Breadcrumb, with HowTo for procedural queries. These work because they mirror how AI answers are built — a clear claim, a source, and an identifiable entity. Markup that contradicts the visible page, fake review schema, or keyword-stuffed descriptions is ignored or erodes trust. Accuracy and a tight match to on-page content matter far more than adding exotic or numerous schema types.
Can I measure whether AI is reading and citing my site?
Yes, using three signals kept separate: AI citations (were you referenced), fetch logs (did AI user agents retrieve your pages), and AI-attributed traffic (did anyone click through). Exposure rarely converts to clicks — Pew data from March 2025 cited by webiano.digital found only 1% of visits with a Google AI summary produced a source click, and Searchless.ai reported on March 24, 2026 that 93% of Google AI Mode searches end without a click. Track citations and fetch logs as your primary KPIs, not visits alone.
Is llms.txt more important than robots.txt or schema?
No — prioritize by certainty. robots.txt matters most because it decides whether AI can read you at all; misconfigure it and you vanish from those answers immediately. Schema.org comes next because it strongly shapes how AI understands your content. llms.txt is a promising but unproven bet on top: cheap and low-risk to publish, but with no major crawler officially committing to it as of April 2026. Fix robots.txt, then schema, then add llms.txt.
LYVIA is a Paris-based AI agency helping US and UK small and mid-size businesses become readable, citable, and visible to AI engines before their competitors are. If you want to know exactly what AI can and cannot read on your site today — and what to fix first — Book a call.
