How to make documentation AI-ready for ChatGPT and Claude

Updated 3 November 2026

To make documentation AI-ready, start with the basics that are proven to matter: AI crawlers must be able to fetch your pages as plain HTML, your robots.txt must let the search bots in, and each article must answer one question clearly, with plan, role and limits stated. Keep pages current, then add Markdown copies, an llms.txt and an MCP server for AI agents. Those extras are cheap, but their measured effect on AI answers is small so far.

This guide is for support leads whose customers now ask ChatGPT, Perplexity or Google's AI Overviews how to use your product, and for teams whose developers point Claude Code or Cursor at your docs. It separates what has evidence behind it from what is still a bet.

What does AI-ready documentation mean?#

AI-ready documentation is documentation that AI systems can fetch, understand and quote correctly: AI-readable docs for answer engines, and docs for AI agents that act on them. Two kinds of reader matter:

• Answer engines (ChatGPT search, Perplexity, Claude, Google AI Overviews and AI Mode) answer your customers' questions and cite pages as sources.
• AI agents (Claude Code, Cursor, Codex, custom assistants) read your docs to do a task, often through Markdown files or an MCP server.

Both are growing fast. Stack Overflow's 2025 survey found 84% of developers use or plan to use AI tools, and Similarweb counted 1.13 billion referral visits from AI platforms in June 2025, up 357% on the year before, with ChatGPT sending over 80% of them. And when Google shows an AI summary, people click through less: Pew Research found users clicked a result link in 8% of visits with an AI summary, against 15% without one. If the answer is shown without the click, the answer had better be right, and it is only as right as your docs. More numbers on where customers look for help are in our customer self-service statistics.

Can AI crawlers actually read your help center?#

Check this first, because nothing else matters if the answer is no.

Is the text in the HTML? Vercel's December 2024 analysis of crawler traffic found that none of the major AI crawlers render JavaScript, Google's Gemini crawler and Apple's being the exceptions. A help center that loads its articles with client-side JavaScript can look empty to ChatGPT's and Claude's crawlers. Test it:

curl -s -A "GPTBot" https://help.example.com/your-article \
  | grep "a sentence from the article"

If the sentence isn't in the response, the crawler doesn't see it either. Server-rendered or prerendered pages fix this.

Do your URLs resolve? In the same Vercel data, about 35% of ChatGPT's and Claude's crawler fetches hit 404 pages. Keep redirects when you move or rename articles, and keep the sitemap current. If you're changing platforms, follow a help center migration checklist.

Is the page public? Answer engines can't read pages behind a login. If you have private docs for customers, accept that AI answers will come from your public pages, forums and third-party sites instead.

Which AI bots should your robots.txt allow?#

Each AI company runs separate bots for training and for search. Blocking "all AI bots" removes you from AI answers too.

The details come from each company's own documentation: OpenAI, Anthropic and Perplexity. A help center that wants to be cited but not used for training can say:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://help.example.com/sitemap.xml

A bot that has its own group ignores the * rules, so repeat any Disallow lines (such as /admin) in each named group. Blocking Google-Extended also stops Gemini Apps from grounding answers in your pages; Google Search and AI Overviews are unaffected.

Also check your CDN or firewall. A "block AI bots" toggle there can override a permissive robots.txt.

How should you write help articles so AI answers correctly?#

Answer engines pull a passage, not your whole site. Each passage has to stand on its own.

• One task per article, named the way customers ask. "Cancel or pause your subscription", not "Subscription management".
• Answer first. The first two sentences under each heading should answer that heading. Our help center article template is built this way.
• State the conditions. Which plan, which role, which platform. "Only admins can delete a workspace" is quotable; "you can delete a workspace" produces wrong answers for everyone else.
• Say what the product can't do. Assistants are often asked "can I…?" If your docs only describe what exists, the model guesses.
• Use one name per thing, and define it once. If "workspace", "account" and "organization" mean the same thing, pick one, and keep a short glossary page that defines your product's terms.
• Give every page clean metadata. A specific title, a one-line description, a last-updated date and, where it applies, the plan or product area. Answer engines and your own search use these to decide which page answers a question.
• Don't hide steps in images or videos. Give every screenshot real alt text, and make sure every step a video shows also exists as text.
• Cover the basics before the edge cases. If you're still building the help center, start with how to build a knowledge base with AI.
• Keep it current. An outdated article is now an outdated answer, repeated with confidence. Our guide to keeping documentation up to date covers a release-by-release routine, and how to auto-update documentation on every release shows how to start it from the release itself.

Google says the same in plainer words: for its AI features, "there are no additional requirements" and no special optimizations needed beyond normal SEO. Its guide also says there's no requirement to break your content into tiny pieces for AI, so don't chop articles into fragments to suit a model.

Should you add llms.txt for documentation?#

It helps less than vendors suggest. llms.txt is a Markdown file at your site's root that lists your most useful pages with short descriptions. Jeremy Howard proposed it in September 2024 as a map for language models.

The evidence so far:

• Google ignores it. Google's guide says you don't need new machine-readable files, AI text files, markup or Markdown to appear in Google Search, and that llms.txt files will neither help nor harm because Search ignores them. Back in 2025, Google's John Mueller compared llms.txt to the keywords meta tag.
• Few bots ask for it. Ahrefs studied 137,210 domains and found that 97% of llms.txt files received zero requests in May 2026. AI retrieval bots made just 1.1% of the requests that did arrive (AI assistants another 2.5%); coding agents such as Claude Code were among the main named readers.

So: add an llms.txt if your docs platform makes it free, especially if developers use your docs with coding agents, but don't expect it to change what ChatGPT says about you. llms-full.txt (all pages in one file) is a common convention, not part of the proposal.

Should your docs be available as Markdown?#

Yes, if it costs you nothing. Agents that fetch a page work better with clean Markdown than with HTML full of navigation and scripts, and the llms.txt proposal suggests serving each page at the same URL with .md added. Three details:

• Mark the Markdown copies noindex (an X-Robots-Tag header works), so search engines don't treat them as duplicates of the HTML pages.
• Keep them generated from the same source as the HTML, so they never drift.
• Let agents find them: the proposal suggests a <link rel="alternate" type="text/markdown"> tag on each HTML page.

How do AI agents use documentation over MCP?#

The Model Context Protocol is an open standard that lets an AI application call tools and read data from other systems. A docs MCP server lets an agent search your docs on demand instead of relying on whatever it learned in training, and, with the right permissions, draft or update pages. For a comparison of the servers available, see MCP servers for documentation.

The same rules apply if you build your own assistant on your docs (retrieval-augmented generation, or RAG): it can only be as accurate as the passages it retrieves. Two cautions: give read-only access to anything customer-facing unless a person reviews writes, and remember that MCP helps agents your users connect themselves. It doesn't change what public answer engines say.

How do you test what AI assistants say about your product?#

Nobody can see inside the models, so test the output. Once a month:

1. Write down 20 real customer questions from tickets and failed help-center searches.
2. Ask each one in ChatGPT, Perplexity, Claude and Google (AI Mode or AI Overviews).
3. Log whether the answer is correct and which page it cites.

4. Fix the article each wrong answer should have come from, then re-ask next month.

To see the traffic side, segment visits by referrer (chatgpt.com, perplexity.ai, claude.ai), and check your server logs for the bot names in the table above (except Google-Extended, which has no user agent of its own). Google counts clicks from AI features in Search Console's Performance report, under the Web search type.

How to make documentation AI-ready: the checklist#

How Clevera help centers are built for AI readers#

Clevera is an AI platform that turns a screen recording into a narrated how-to video, a step-by-step help article and a page on your hosted help center, in 74 languages. Its hosted help centers handle most of the technical half of the checklist, on every plan including Free:

• Prerendered pages and a sitemap, so the article text is in the HTML crawlers fetch.
• llms.txt, llms-full.txt and a .md copy of every page, under the setting llms.txt and Markdown copies ("Lets AI tools read your docs"), which is on by default. On our own help center the Markdown copies are served with noindex. Private help centers (Pro and up) keep all of this behind the sign-in.
• Ask AI for your readers (turn on Show the assistant in Settings → AI assistant; 300 answers a month on Free). The built-in assistant answers from your published pages only, links its sources, says so when the docs don't cover a question, and lists those questions under Unanswered questions so you know what to write next (our guide to help center content gaps shows how to work through them).
• AI agents and MCP (Pro and up). Claude Code, Codex, Cursor, Claude or ChatGPT can search, draft and, with publishing rights, update your help center through Clevera's MCP server and API. A Support assistant token can only search published, public pages.

This is the top of the llms.txt our own help center publishes (shortened):

# Clevera Help Center

> Clevera turns screen recordings into polished videos: record your screen
> or let the AI record a website, edit with zooms, captions and AI
> voice-overs, then share or embed the result. ...

Every page is also available as Markdown: append `.md` to its URL.
The full text of all pages is at https://help.clevera.ai/llms-full.txt.

### Getting started

- [Understand what Clevera makes](https://help.clevera.ai/docs/getting-started/what-clevera-makes.md): ...

The writing half is still yours, and Clevera helps there too: articles generated from a screen recording come out as numbered steps with screenshots of what's on screen, and the recording behind a video can also produce its written article, so steps don't live only in a video. No platform can promise that ChatGPT will cite you, and we don't either.

Start free with a hosted help center, or import your current one from Zendesk, Intercom, GitBook and other platforms. Settings are in Control who can read a doc site, Set up the doc site assistant and Connect AI agents to your doc sites. The product overview is on the help center software page and in our doc sites launch post.

FAQ#

Is llms.txt worth adding to a help center?#

Add it if your platform generates it for free. It costs nothing and coding agents do read it, but Google doesn't use it and studies show almost no requests from AI search bots. Crawlable HTML and clear, current articles matter far more.

How do I get ChatGPT to answer questions about my product correctly?#

Make sure OAI-SearchBot can crawl your public help center, that article text is in the HTML, and that each article answers one question with the conditions stated. Then test real customer questions monthly and fix the pages behind wrong answers. This improves the odds; no one can make ChatGPT cite you.

Should I block GPTBot and ClaudeBot?#

That's a training decision, not a visibility one. Blocking GPTBot and ClaudeBot keeps your docs out of model training; it doesn't remove you from ChatGPT or Claude search, which use OAI-SearchBot and Claude-SearchBot. Don't block those if you want to be cited.

Does structured data help AI Overviews?#

Google says no special schema is needed for AI features. Use structured data where it describes visible content, as you would for normal search.

What's the difference between AI-ready docs and an AI chatbot on my help center?#

AI-ready docs help outside assistants answer from your pages. A help-center chatbot, such as Clevera's Ask AI, answers your readers on your own site from the same pages. Both depend on clear, current articles.