# Why llms.txt matters: making your website discoverable by AI - agentmarkup > Search is changing. LLMs now answer questions by synthesizing web content. llms.txt gives them a structured overview of your site. Learn why it matters and how to add it. Source: https://agentmarkup.dev/blog/why-llms-txt-matters/ By [Sebastian Cochinescu](/authors/sebastian-cochinescu/) · March 20, 2026 · 6 min read # Why llms.txt matters: making your website discoverable by AI Search is changing. LLMs like ChatGPT, Claude, and Perplexity now answer questions by synthesizing information from the web. If your website is not structured for these systems to understand, you are invisible to a growing share of how people find information. ## The shift from search engines to AI answers For two decades, the path to online visibility was clear: optimize for Google, get ranked, get clicked. That model is not going away, but it is being joined by a second discovery channel. When someone asks ChatGPT "what is the best leather wallet under $100" or asks Perplexity "how do I add structured data to my website," the answer comes from synthesized web content, not a list of blue links. The question for website owners is no longer just "can Google index my pages?" but also "can an LLM understand what my site is about?" ## What is llms.txt? [llms.txt](https://llmstxt.org) is a proposed standard that gives LLMs a structured overview of your website. It is a plain text file served at `/llms.txt`, written in simple markdown, that describes your site name, purpose, and pages. Think of it as robots.txt for AI understanding. While robots.txt tells crawlers what they can access, llms.txt tells them what your site actually is. ``` # My Shop > Handcrafted leather goods since 2015. ## Products - [Wallets](https://myshop.com/products/wallets): Full-grain leather bifold wallets - [Bags](https://myshop.com/products/bags): Handmade leather messenger bags ## Support - [FAQ](https://myshop.com/faq): Common questions about orders and shipping - [Contact](https://myshop.com/contact): Get in touch with our team ``` ## Why not just let AI read your HTML? HTML pages are designed for humans. They contain navigation menus, cookie banners, ad scripts, analytics tags, and layout markup that has nothing to do with your actual content. An LLM reading your raw HTML has to guess what matters and what is noise. llms.txt gives the LLM a clean, noise-free map of your site. No guessing required. It is the equivalent of handing someone a table of contents instead of making them skim every page. ## Is llms.txt actually used today? Honestly, adoption is early. Perplexity has shown interest in consuming llms.txt. Other major AI systems have not publicly committed to it. The format is a proposal from llmstxt.org, not an official standard. But the cost of generating it is near zero. A build-time plugin like [agentmarkup](https://github.com/agentmarkup/agentmarkup) can produce it automatically from your existing site config. The downside risk is an extra text file in your build output. The upside is being ready when AI systems start looking for it. ## The three layers of machine readability llms.txt is one piece of a larger picture. A truly machine-readable website has three layers: - **llms.txt** - a high-level map of what your site is and what pages exist - **JSON-LD structured data** - page-level metadata telling search engines and AI what each page represents (a product, an article, a FAQ) - **robots.txt AI crawler rules** - explicit permissions for which AI bots can access your content Each layer serves a different purpose. llms.txt is for AI discovery. JSON-LD is for search engine rich results. robots.txt is for access control. Together, they make your site understandable by both traditional search engines and the new generation of AI systems. ## How to add llms.txt to your site You can write llms.txt by hand, but keeping it in sync with your actual pages is tedious. A build-time tool does it automatically. With agentmarkup, you configure your site structure once and the plugin generates llms.txt, injects JSON-LD, and patches robots.txt on every build: ``` pnpm add -D @agentmarkup/vite # or @agentmarkup/astro or @agentmarkup/next ``` Read the [full llms.txt guide](/docs/llms-txt/) for configuration details. ## The bottom line The web is getting a second audience. Humans still matter, but AI systems are increasingly the first reader of your content. llms.txt is a low-cost, low-risk way to make sure those systems understand what you have built. Whether or not it becomes a formal standard, the practice of structuring your site for machine readability is here to stay. ## Make your website machine-readable agentmarkup is an open-source build-time toolkit for Vite, Astro, and Next.js that generates llms.txt, injects JSON-LD structured data, creates optional markdown mirrors from final HTML when raw pages need a cleaner agent-facing fetch path, manages AI crawler robots.txt rules, patches optional Content-Signal and canonical mirror headers, and validates everything at build time. Zero runtime cost. Learn more GitHub ``` pnpm add -D @agentmarkup/vite # or @agentmarkup/astro or @agentmarkup/next ``` Written by [Sebastian Cochinescu](/authors/sebastian-cochinescu/) · Developer of agentmarkup Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix. ## More from the blog ### How to add llms.txt, JSON-LD, and AI crawler controls to Next.js Use @agentmarkup/next to generate llms.txt, inject JSON-LD, manage AI crawler rules, and understand the dynamic SSR boundary in Next.js. March 23, 2026 · 8 min read ### When markdown mirrors help, and when they do not A practical guide to when generated markdown mirrors add signal, when HTML is already enough, and how to avoid unnecessary downsides. March 20, 2026 · 7 min read ### Is your website ready for AI? Free LLM discoverability checker Audit your website for llms.txt, JSON-LD, robots.txt, markdown mirrors, and sitemap. Free tool for e-commerce and brand websites. March 20, 2026 · 8 min read ### Build-time markdown mirrors for agent readability: Cloudflare comparison Build-time markdown generation for AI readability, including when it helps and how it compares to Cloudflare runtime extraction. March 20, 2026 · 7 min read ### How to make your brand appear in AI conversations Organization schema, llms.txt, and FAQ markup make your brand visible in ChatGPT, Claude, and Perplexity answers. March 20, 2026 · 7 min read ### Why LLM-optimized e-commerce websites sell more Product JSON-LD, llms.txt, and AI crawler access make your store visible in AI product recommendations. March 20, 2026 · 8 min read ### Every AI crawler indexing your website in 2026 Complete list: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and more. What each does and how to control access. March 20, 2026 · 8 min read ### JSON-LD structured data: the complete guide for web developers Schema types, JSON-LD vs microdata, common mistakes, and build-time validation. March 20, 2026 · 10 min read ### What is GEO? Generative Engine Optimization explained for developers What is real, what is hype, and what you can do today to make your site citeable by AI. March 20, 2026 · 7 min read