llms.txt: What It Is, Who Actually Reads It, and What to Do Instead

Quick answer: llms.txt is a proposed file, not a standard. Jeremy Howard put it forward in September 2024, and it sits at your domain root as a Markdown index of your most useful pages so an AI tool can find them without crawling everything. The part most agencies skip: as far as anyone can verify, no major AI provider has committed to reading it, and Google has said publicly that it does not use it. Ahrefs found 97% of the llms.txt files it examined were never requested at all. It costs almost nothing to publish, so it is no scandal, but it is not what makes your site visible in AI answers either. What does work is mundane: correct robots.txt rules for the AI crawlers that matter, server rendered content a crawler can read, documented structured data, and a page hierarchy that makes sense. This guide covers the specification, each provider’s verified position with dates, a copyable file for a Malaysian service business, and a decision box.

What llms.txt proposes, and who wrote it

The proposal was published by Jeremy Howard at llmstxt.org on 3 September 2024. The version live when we checked on 15 October 2026 is v2, last modified 10 August 2026.

The problem it describes is real. Language models work with a limited context window, and a modern web page wraps its useful content in navigation, scripts, cookie notices and markup, so a tool that fetches your page spends much of its budget on furniture. The answer proposed is a short Markdown file pointing an AI tool straight at the pages worth reading.

Two things follow, and both get lost in the marketing. First, it was designed for the moment an AI tool is already looking at your site, because a user sent it there or a developer pointed an agent at your documentation. It was never a discovery or ranking mechanism, and the proposal does not claim to be one. Second, it is a proposal: no standards body, no registration, no requirement on anyone to honour it. robots.txt became a convention because crawler operators chose to follow it over decades, and llms.txt has not had that happen yet.

The file format

The specification sets out a required order. Quoting the structure:

1. An optional byte-order mark
2. An H1 with the name of the project or site. This is the only required section
3. A blockquote with a short summary of the project, containing key information necessary for understanding the rest of the file
4. Zero or more Markdown sections of any type except headings
5. Zero or more sections delimited by H2 headers, containing “file lists” of URLs

Each item in a file list is a Markdown link, [name](url), optionally followed by a colon and notes about that page.

One H2 has a reserved meaning. The specification says the “Optional” section “is used, by convention, for secondary information: links an agent can skip when a shorter context is needed”. If you have pages that matter but are not essential, that is where they go.

llms.txt and llms-full.txt are not the same thing, and only one is in the spec

Vendors blur this, so it is worth getting right. llms.txt is an index: links with short descriptions, kept small. llms-full.txt is a single file holding the full text of the documentation, concatenated, and it is not defined in the specification at all. We checked the spec on 15 October 2026 and it describes no expanded variant. llms-full.txt is a convention that grew out of documentation tooling and spread as developer platforms adopted it.

The size difference is the point. On 15 October 2026 we fetched both files from Anthropic’s developer documentation:

File	Size
docs.anthropic.com/llms.txt	about 85 KB
docs.anthropic.com/llms-full.txt	about 44 MB

An index of 85 KB is something a tool can read in one pass. A 44 MB text file is not: it exists so a developer or a build process can pull a whole corpus deliberately. For a Malaysian service business with thirty pages, llms-full.txt has no sensible purpose. Do not let anyone bill you for one.

Where the file goes

At the root, https://yourdomain.com/llms.txt.

The specification also allows it at any path, covering the pages beneath that path, so https://yourdomain.com/docs/llms.txt would cover the documentation section. Where more than one file applies, the spec says agents should use the most specific one. For almost every Malaysian business, one file at the root is all that is relevant.

The honest part: which AI crawlers actually read it

This is the section that decides whether the file is worth your time, so here is what can be verified, with dates and sources, and what cannot.

Provider	Verified position	Source and date
Google	Does not use it. John Mueller wrote “FWIW no AI system currently uses llms.txt” on Bluesky on 17 June 2025. Google’s own AI features documentation, updated 10 December 2025, says “You don’t need to create new machine readable files, AI text files, or markup to appear in these features”	Mueller via Search Engine Roundtable, 18 June 2025; Google Search Central AI features page
Google (Chrome team)	Different answer, different product. Chrome’s Lighthouse has an llms.txt audit under agentic browsing, last updated 5 May 2026. It treats the file as optional, marks it Not Applicable on a 404, and frames the benefit as agents spending less time working out site structure. It makes no claim about Google Search	developer.chrome.com Lighthouse docs
OpenAI	No commitment found to consuming llms.txt. OpenAI’s crawler documentation describes OAI-SearchBot, GPTBot and ChatGPT-User and says nothing about llms.txt as an input. OpenAI’s developer docs do publish an llms.txt as an index of their own documentation, which is the opposite direction of travel	developers.openai.com bots documentation
Anthropic	No commitment found to consuming llms.txt. Anthropic’s crawler guidance documents ClaudeBot, Claude-User and Claude-SearchBot and explains robots.txt control, with no mention of llms.txt. Anthropic’s documentation site does publish llms.txt and llms-full.txt	Anthropic support documentation; files fetched directly
Perplexity	No commitment found to consuming llms.txt. Perplexity’s crawler guidance documents PerplexityBot and Perplexity-User without mentioning llms.txt. Its documentation site publishes one as a doc index	docs.perplexity.ai

Notice the pattern across OpenAI, Anthropic and Perplexity. Each of them publishes an llms.txt for its own developer documentation, because that is a genuinely useful thing to hand a coding agent. None of them documents reading yours when answering a user’s question. Publishing is not consuming, and claims that “OpenAI and Anthropic support llms.txt” almost always rest on this confusion.

What the measurement says

Ahrefs published an analysis by Ryan Law examining 137,000 domains. Of roughly 38,000 sites carrying a valid llms.txt, 97% of those files were never requested at all during the month studied, May 2026. His conclusion: no evidence llms.txt improves AI retrieval, traffic or model accuracy, no provider committed to parsing it, and the idea is a solution in search of a problem.

Separate log analyses through 2026 point the same way, with llms.txt requests a negligible share of activity from GPTBot, ClaudeBot and PerplexityBot, which crawl HTML directly. We have not independently reproduced those studies, and say so rather than presenting them as our own finding.

What we cannot verify

Being straight about the edges of the evidence:

• We cannot prove a negative. No provider publishing a commitment is not the same as no provider ever reading the file. Internal behaviour is not observable from outside, and absence of documentation is weaker evidence than a documented refusal
• Claims that Anthropic, Cursor or Mintlify “officially support” llms.txt circulate widely, and we found no primary documentation of consumption by any of them. What we could verify is that they publish the file. Treat the stronger claim as unsourced
• Positions change. Every date above matters. If a provider announces consumption next quarter, this section is out of date, not wrong

The one place the file genuinely does something

There is one narrow, real use case, worth naming because it stops the discussion being purely negative. If your site is technical documentation, a developer platform or an API reference, and your users point coding agents at it, an llms.txt index helps that agent find its way without crawling everything. Chrome’s Lighthouse audit exists for exactly this scenario, and it is why the developer platforms publish one.

If your site is a Malaysian law firm, a clinic, a manufacturer or a retailer, this does not apply. Nobody is pointing an agent at your site to build against it.

What does work instead

This is the part worth your budget. None of it is exotic.

1. robots.txt rules for the AI crawlers that matter

robots.txt is honoured in practice, which is the whole difference. The mistake we see most often is a site that allows one or two AI crawlers and has never heard of the others, including the ones that actually feed answers to users.

The agents worth knowing, all from the providers’ own documentation, checked 15 October 2026:

Agent	Provider	What it does	Obeys robots.txt?
GPTBot	OpenAI	Crawls content that may be used to train foundation models	Yes
OAI-SearchBot	OpenAI	Surfaces websites in ChatGPT’s search features	Yes
ChatGPT-User	OpenAI	Fetches a page because a user action in ChatGPT asked for it	Not subject to robots.txt, being user initiated
ClaudeBot	Anthropic	Collects web content to improve the models	Yes
Claude-SearchBot	Anthropic	Navigates the web to improve search result quality	Yes
Claude-User	Anthropic	Accesses sites when a user asks Claude a question	Yes, per Anthropic’s guidance
PerplexityBot	Perplexity	Surfaces and links sites in Perplexity results. Explicitly not used for training foundation models	Yes
Perplexity-User	Perplexity	Visits a page to answer a specific user question	Generally ignores robots.txt, being user initiated
Google-Extended	Google	Controls use of your content for Gemini and related model training. It is not a crawler and does not affect Google Search	Yes, as a robots.txt token

The decision that follows is a business one. The search and user facing agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are how your business gets mentioned in an AI answer. The training crawlers (GPTBot, ClaudeBot, Google-Extended) improve a model with no citation in return.

A reasonable default for a Malaysian business that wants to be found:

# Allow the agents that can cite you
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Decide deliberately on training crawls
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

Sitemap: https://yourdomain.com/sitemap_index.xml

If you would rather not contribute to model training, change those last two to Disallow: / and add User-agent: Google-Extended with Disallow: /. Blocking the search agents instead is the one choice that reliably costs you visibility. Our piece on Cloudflare’s AI crawler update covers what happens when the decision is made at the network layer rather than in robots.txt.

2. Server side rendering, so the content is actually there

An AI crawler that fetches your page and receives an empty shell waiting for JavaScript gets nothing. This is the most common reason a site is invisible to AI tools, and it has nothing to do with llms.txt. Test it bluntly, from the command line:

curl -s https://yourdomain.com/your-service-page/ | wc -c

If the body text of your service page is not in that response, no file at your domain root will fix it. The fix is server side rendering, static generation or prerendering, and it is a development task. Our technical SEO guide covers the diagnosis, and AI friendly websites covers the build decisions.

3. Structured data, within the documented list

Use the types Google documents, chiefly Organization, LocalBusiness, BreadcrumbList and Article, and make the markup agree with what the page visibly says. Google’s AI features documentation, updated 10 December 2025, states plainly that there is “no special schema.org structured data that you need to add”, so treat any pitch for “AI schema” with suspicion. Documented markup is worth using because it makes your facts unambiguous, and unambiguous facts are harder to get wrong about you.

4. A content hierarchy that holds up

What llms.txt tries to solve with a file, a sensible site solves by being sensible: one page per question, headings that match the questions, a direct answer near the top, breadcrumbs, a working XML sitemap, and no near duplicate pages competing with each other. That structure helps a crawler, an AI tool and a human reader at once, and unlike llms.txt it is honoured by everything.

For the wider programme, see answer engine optimisation, how to get cited by AI and AI search readiness. To measure any of it, how to track traffic from AI Overviews sets out what is and is not visible.

A copyable llms.txt for a Malaysian service business

If you have decided to publish one, keep it short, keep it accurate, and keep it maintained. Here is a complete file for a fictional Kuala Lumpur accounting firm. Replace the names and URLs.

# Tan & Rahim Chartered Accountants

> A chartered accounting firm in Kuala Lumpur providing audit,
> tax and corporate secretarial services to Malaysian SMEs.
> Registered with the Malaysian Institute of Accountants.
> Offices in KL and Penang. Serving clients Malaysia wide.

Fees depend on turnover, transaction volume and whether the
engagement includes audit. All quotes are issued in writing
after a scoping call.

## Services

– [Statutory audit](https://example.com.my/statutory-audit/): Audit of company accounts for Companies Act filing.
– [Corporate tax](https://example.com.my/corporate-tax/): Tax computation, estimates and filing for Sdn Bhd companies.
– [Personal tax](https://example.com.my/personal-tax/): Individual and partnership tax filing.
– [Corporate secretarial](https://example.com.my/corporate-secretarial/): Company incorporation, annual returns and registry filings.
– [Payroll and SST](https://example.com.my/payroll-sst/): Monthly payroll, statutory contributions and SST returns.

## Guides

– [Filing deadlines for Sdn Bhd companies](https://example.com.my/sdn-bhd-filing-deadlines/): What is due, when, and the penalties for late filing.
– [When a company must be audited](https://example.com.my/audit-exemption/): Audit exemption thresholds and who still needs an audit.
– [Choosing an accounting firm](https://example.com.my/choosing-an-accountant/): What to ask before appointing one.

## About

– [About the firm](https://example.com.my/about/): History, partners and statutory registrations.
– [Our team](https://example.com.my/team/): Partners and managers, with qualifications.
– [Contact](https://example.com.my/contact/): Office addresses, phone numbers and enquiry form.

## Optional

– [Client portal](https://example.com.my/portal/): Login for existing clients.
– [Careers](https://example.com.my/careers/): Current vacancies.

What makes that file worth having rather than noise: the H1 names the organisation, the blockquote states what the firm does and where in a form a model can quote, every link describes what is on the page, nothing is listed twice, and low value pages sit under Optional where the spec intends them.

What to avoid: dumping in your whole sitemap, pasting article excerpts as descriptions, listing pages that redirect or no longer exist, and publishing once then never touching it again. A stale file is worse than none, because the few tools that read it read something false.

Your site may already have one, and it may be a mess

Check before you commission anything. Several WordPress SEO plugins now generate llms.txt automatically, and the output is often nothing like the specification intends.

We checked mediaplusdigital.com.my/llms.txt on 15 October 2026. It exists, generated by All in One SEO, and runs to about 26 KB across 114 lines. Instead of short descriptions it carries the opening paragraph of each post, so the file is mostly prose written for human readers, and the H1 is the site’s SEO title rather than the organisation’s name. It is not harmful. It is also not doing the job the specification describes, and nobody would have known without opening it.

Three quick checks:

• Fetch yourdomain.com/llms.txt. Does it exist, and did you know?
• If it exists, read it. Is the H1 your organisation’s name? Do the descriptions describe pages, or are they article excerpts?
• Do all the URLs in it return 200? A plugin generated file will not notice a page you deleted

Should you bother? A decision box

Your situation	Our recommendation
You publish technical documentation or an API that developers point coding agents at	Yes. This is the use case the file was designed for. Publish a hand written index and keep it current
You have a WordPress site and a plugin already generates one	Tidy it or turn it off. Ten minutes of review. Either make it match the spec or remove it, rather than leaving something inaccurate at your root
You are a Malaysian service business and someone is quoting you a project fee for llms.txt	No. Publishing the file is twenty minutes of work with no evidence of benefit. Put the budget into rendering, hierarchy and content
Your site is invisible in AI answers and you want that fixed	No, not this. Start with whether your content is server rendered and whether your robots.txt allows the search agents. That is where the problem usually is
You want it for completeness, and your developer can do it today	Fine. It is cheap and does no harm. Just do not count it as your AI visibility strategy, and diarise a review when you next change your page structure

The summary in one line: a correct llms.txt is a tidy piece of housekeeping, and anyone selling it as essential to AI visibility is either behind the evidence or counting on you not checking.

Frequently asked questions

Is llms.txt an official standard?

No. It is a proposal published by Jeremy Howard in September 2024, now at v2, with no standards body behind it and no obligation on anyone to honour it.

Does Google use llms.txt?

No. John Mueller said on 17 June 2025 that no AI system then used it, and Google’s AI features documentation, updated 10 December 2025, says you do not need new machine readable files or AI text files to appear in those features. The separate Lighthouse audit from the Chrome team is about agentic browsing, not Google Search.

Will publishing one hurt my site?

There is no evidence of harm. The two real risks are maintenance, because a stale file states things that are no longer true, and convenience for scrapers, since you hand over a tidy index of your best content.

What is the difference between llms.txt and robots.txt?

Different jobs, different standing. robots.txt tells crawlers what they may access and is widely honoured; llms.txt suggests what is worth reading and is largely unread. If you do one, do robots.txt properly.

Do I need llms-full.txt as well?

Almost certainly not. It is not in the specification, and in practice it is a full text dump of a documentation corpus. Anthropic’s runs to roughly 44 MB. For a business site with a few dozen pages it serves no purpose.

My SEO plugin created an llms.txt. Should I keep it?

Open it first. Auto generated files commonly use the site’s SEO title as the H1 and paste article excerpts as descriptions, which is not what the spec describes. Either tidy it into a proper index or disable the feature. Leaving an inaccurate file at your root is the worst option.

If llms.txt does not work, how do I get into AI answers?

Be readable and be worth citing. Server render your content, allow the search and user facing AI agents in robots.txt, structure pages so one page answers one question with the answer near the top, use documented structured data, and earn mentions on sites that AI tools already draw from. See how to get cited by AI and AI Overview citations.

Should I block AI crawlers entirely?

Only with the distinction in mind. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended) keeps your content out of model training and costs you nothing in citations. Blocking the search and user facing agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot) removes you from AI answers. Most businesses that block everything did not intend the second effect.

The bottom line

llms.txt is a sensible idea that has not been adopted. The specification is clear, the file is easy to write, and if your site is developer documentation it does a real job. For a Malaysian service business the evidence says it is housekeeping: Google has said it does not use it, no major provider documents consuming it, and the largest published analysis found 97% of these files were never requested.

That is not a reason to be cynical about AI visibility, but a reason to spend on the parts that are honoured. Your robots.txt is read by every crawler that matters, and most sites have it half right. Your content either renders on the server or it does not, and that single fact decides whether an AI tool can see you at all. Your hierarchy either answers one question per page or scatters the answer across four competing ones. Fix those and llms.txt becomes a twenty minute afterthought, which is the correct size for it.

If you would rather have this assessed than argued about, our GEO services in Malaysia cover AI visibility work from crawler access through to the content that gets cited, and our SEO services in Malaysia cover the search foundation underneath it. Get in touch for a free check of your robots.txt, your rendering and whatever is currently sitting at your llms.txt.