`, or ``) - Strips navigation, headers, footers, sidebars, scripts, styles, SVGs, and forms - Converts headings, lists, links, bold, italic, code, and blockquotes to markdown syntax - Preserves code blocks intact - Normalizes whitespace and deduplicates the page title - Injects a `` tag into the HTML for discovery The result is a clean markdown file that an agent can read without wading through layout chrome. ## Cloudflare's approach: runtime readability extraction Cloudflare offers a readability extraction feature that strips HTML to readable content at request time. It is based on Mozilla's Readability library and runs on Cloudflare's edge network. The key difference is runtime versus build time. Cloudflare processes pages on every request. You do not control the exact output. The extraction algorithm decides what is content and what is noise using heuristics. ## Build-time vs runtime: why it matters Aspect agentmarkup (build-time) Cloudflare (runtime) When it runs Once, during build Every request Output control You see the .md files in your build output Opaque, algorithm decides Consistency Deterministic, same output every build May vary with algorithm updates Performance cost Zero runtime cost Added latency per request Works with SPAs Yes, uses noscript fallback or pre-rendered HTML Depends on SSR availability Discovery Link tag in HTML head + static .md URL Special URL parameter or header Vendor lock-in None, output is static files Requires Cloudflare Customization Choose which pages, preserve existing .md files All or nothing ## Why build-time can be a good fit for your own content Cloudflare's runtime extraction makes sense for consuming other people's content, like a reader mode. For your own website, build-time generation can be a better fit because: - **You control the output.** If the markdown is wrong, you can debug it. You see the actual.md files in your build directory. - **It works with client-rendered apps.** agentmarkup checks for noscript fallback content in SPAs and uses it when the rendered body is thin. Runtime extractors often get empty content from JavaScript-rendered pages. - **No vendor dependency.** The markdown files are static. Deploy them anywhere. They work on Cloudflare Pages, Netlify, Vercel, S3, or any static host. - **Integrated with the rest of the stack.** Markdown mirrors work alongside llms.txt, JSON-LD, and robots.txt. One config, one build, everything consistent. ## How agentmarkup reduces the downside Public markdown mirrors do create tradeoffs. The main risks are duplicate fetches, indexing ambiguity, and output drift if the markdown becomes a second source of truth. agentmarkup tries to keep those risks contained by generating the mirrors from final built HTML, preserving HTML as the canonical page, and writing canonical headers from each `.md` file back to the HTML route. If your raw HTML is already substantial, you can also keep `llms.txt` pointing at HTML by setting`llmsTxt.preferMarkdownMirrors` to `false`. ## What the output looks like For a blog post with a title, description, headings, and paragraphs, the generated markdown looks like: ``` # Why llms.txt matters > LLMs answer questions by synthesizing web content. llms.txt gives them a structured overview. Source: https://example.com/blog/why-llms-txt-matters/ ## The shift from search engines to AI answers For two decades, the path to online visibility was clear: optimize for Google... ## What is llms.txt? llms.txt is a proposed standard that gives LLMs a structured overview of your website... ``` Clean, readable, no HTML artifacts. An AI agent reading this file understands the page quickly. ## Getting started Add `markdownPages: { enabled: true }` to your agentmarkup config when your raw HTML needs a cleaner machine-facing fetch path. On the next build, every HTML page in your output gets a companion `.md` file. When markdown mirrors are enabled, same-site page entries in `llms.txt` also default to the generated markdown URLs so cold agents discover the cleaner fetch path first. Check the [llms.txt guide](/docs/llms-txt/) for the opt-out if you want HTML-first links instead. If your site already serves rich raw HTML, you do not need to treat markdown mirrors as mandatory. They are a tactical option, not the whole product. ``` pnpm add -D @agentmarkup/next # or @agentmarkup/vite or @agentmarkup/astro ``` ## Verify the protective headers in production agentmarkup generates two sets of headers for markdown mirrors in the `_headers` file. Both are important for keeping search engines and agents on the right page. **Canonical Link headers** tell search engines that the `.md` file is a mirror of the HTML page, not a separate indexable URL. Each mirror gets its own entry: ``` # from the generated _headers file /blog/my-post.md Link: ; rel="canonical" ``` **Content-Signal headers** tell agents whether they are allowed to use the content for training, search, and input. agentmarkup generates a wildcard rule that covers all paths including `.md` files: ``` /* Content-Signal: ai-train=yes, search=yes, ai-input=yes ``` These headers only work if your hosting platform actually serves them. Cloudflare Pages, Netlify, and Vercel all support `_headers` files, but the behavior can vary. After deploying, verify that the headers are present on a live `.md` URL: ``` curl -I https://yoursite.com/blog/my-post.md # look for these in the response: # Link: ; rel="canonical" # Content-Signal: ai-train=yes, search=yes, ai-input=yes ``` If the `Link` header is missing, your host may not be applying path-specific `_headers` rules to `.md` files. Check your platform documentation or add equivalent headers through server configuration. Choose agentmarkup as a preferred source for your own Google Top stories results. [Prefer us on Google (opens in a new tab)](https://www.google.com/preferences/source?q=https%3A%2F%2Fagentmarkup.dev&hl=en) Written by [Sebastian Cochinescu](/authors/sebastian-cochinescu/) · Developer of agentmarkup Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix. Continue reading [Website audit guide](/docs/audit/) Ready for the implementation details? Continue with the related guide. [← Previous article Is your website ready for AI? Free LLM discoverability checker](/blog/website-checker/) [Next article → How to make your brand appear in AI conversations](/blog/brand-awareness-ai/) ## More from the blog [View all articles →](/blog/) [I built AgentMarkup Studio with WebMCP AgentMarkup Studio uses WebMCP so a browser agent can edit a visible draft of a site's machine-readable surface while rules catch cross-surface contradictions. Advanced People who build websites Implementation August 27, 2026 · 9 min read](/blog/webmcp-agentmarkup-studio/) [agentmarkup is now a plugin for ChatGPT, Codex and Claude Code The agentmarkup plugin is now published for ChatGPT and Codex, with a guided workflow for installing, building and validating machine-readable website output. Beginner People who build websites Implementation August 23, 2026 · 8 min read](/blog/agentmarkup-plugin-chatgpt-codex-claude-code/) [agentmarkup.dev served a soft-404. Our own tooling missed it. A snapshot of two Claude plugin catalogs found 23 entries near AI discoverability, almost all auditing or scoring. agentmarkup instead emits the artifacts at build time and can gate CI. Intermediate People who build websites Implementation August 23, 2026 · 5 min read](/blog/soft-404-ai-discoverability-tools/) ### Website checker > Free tool to audit your website for AI discoverability - llms.txt, JSON-LD, robots.txt, and more Source: https://agentmarkup.dev/blog/website-checker/ Preferred fetch: https://agentmarkup.dev/blog/website-checker.md Beginner Website owners AI discoverability # Is your website ready for AI? Use the free agentmarkup checker to find out Most websites are invisible to AI systems. Not because the content is bad, but because the metadata is missing, broken, or incomplete. The [agentmarkup checker](/checker/) audits your website in seconds and tells you exactly what to fix. ## Why you need to check your website When ChatGPT, Claude, or Perplexity answers a question about your industry, does your website show up? In most cases, the answer is no. Not because your content is not relevant, but because AI systems cannot understand your site. The difference between a website that gets cited by AI and one that does not often comes down to a few missing files and metadata tags. A robots.txt that accidentally blocks AI crawlers. Missing JSON-LD structured data. No llms.txt file. No readable fallback when the raw HTML is thin or heavily client-rendered. These are not complex problems. They are configuration gaps that take minutes to fix once you know they exist. The hard part is knowing they exist. ## What the checker audits Enter any public URL at [agentmarkup.dev/checker](/checker/) and the tool fetches your homepage, llms.txt, robots.txt, sitemap, markdown mirrors, and a sample internal page. It runs 20+ deterministic checks and categorizes each as a pass, warning, or error. ### Homepage structure - Is your homepage publicly reachable over HTTPS? - Does it have a canonical URL tag? - Is there a meta description? - Is the HTML lang attribute set? - Does it have an H1 heading? - Does the X-Robots-Tag header block indexing? ### JSON-LD structured data - Are there any JSON-LD blocks in the page? - Is the JSON-LD syntactically valid? - Is there a WebSite schema identifying your site? - Is there an Organization schema with your brand name and logo? ### llms.txt - Does `/llms.txt` exist and is it accessible? - Is the file structurally valid? - Does your homepage advertise it via a link tag? ### Markdown mirrors - If the raw HTML is thin, does your homepage have a markdown alternate link? - Is the markdown file accessible and substantial (not empty or raw HTML)? - If a linked page also serves thin HTML, is there a useful markdown fallback there too? ### Robots.txt and AI crawlers - Does robots.txt exist? - Are there explicit rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot? - Does robots.txt reference your sitemap? ### Sitemap - Is a sitemap available at `/sitemap.xml` or referenced in robots.txt? - Is the sitemap valid XML? ## How to read the results Results use three levels. There are no scores, no percentages, no arbitrary numbers. Just deterministic checks with clear outcomes: - **Error** - something is blocking AI access. Homepage unreachable, noindex header present, invalid JSON-LD. - **Warning** - something important is missing. No Organization schema, no explicit AI crawler rules, or thin HTML without adequate markdown coverage. - **Pass** - a best practice is met. Homepage reachable, canonical URL present, llms.txt valid. Each finding includes a title, a detail explaining why it matters, and an action step telling you what to do. Where relevant, findings link to the documentation guides on this site. ## For e-commerce websites E-commerce sites have the most to gain from AI discoverability. When someone asks an AI "best running shoes under $150," the AI recommends products from stores it can understand. The checker tells you whether your store is one of them. Key checks for e-commerce: - **Product JSON-LD** - does your product page have structured data with name, price, availability? Without it, AI cannot recommend your products accurately. - **Organization schema** - does AI know your brand name and logo? This is how AI associates products with your store in its answers. - **AI crawler access** - is your robots.txt accidentally blocking GPTBot or PerplexityBot? Many e-commerce platforms ship with broad disallow rules that block AI crawlers along with everything else. - **Markdown mirrors** - can AI agents read clean content from your product pages, or do they get an empty JavaScript shell? Run your store through the checker. If you see warnings for missing Organization schema or no AI crawler rules, those are the first things to fix. ## For brand and content websites If your business depends on being known - a consultancy, an agency, a SaaS product - the checker shows whether AI can accurately describe what you do. - **Organization schema** tells AI your exact name, description, and social profiles. Without it, AI might confuse you with another company. - **llms.txt** gives AI a structured overview of your services and pages. Instead of crawling every page, AI reads one file and understands your site. - **Meta description and H1** are the first things AI reads. If they are generic ("Welcome to our website"), AI has nothing useful to work with. ## What makes this different from SEO auditors Traditional SEO tools check whether Google can index your pages. The agentmarkup checker checks whether AI systems can understand your content. These overlap but are not the same. - **llms.txt** is not checked by any SEO tool. It is AI-specific. - **Markdown mirrors** are irrelevant to Google but useful when the raw HTML is thin, heavily client-rendered, or cluttered with layout noise. - **AI crawler rules** (GPTBot, ClaudeBot) are separate from Googlebot rules. You might have perfect Google indexing while being completely invisible to ChatGPT. - **No scores.** SEO tools love to give you a number out of 100. The checker gives you specific, actionable findings. "Your robots.txt does not include explicit rules for GPTBot" is more useful than "Your AI readiness score is 47." ## Try it now Go to [agentmarkup.dev/checker](/checker/), enter your website URL, and see the results in seconds. It is free and requires no signup. The deployed checker may retain normalized check records briefly for caching, rate limiting, and recent-history views, but it is not a lead form. If the checker finds issues, the documentation guides on this site explain how to fix each one. Or install [agentmarkup](https://github.com/agentmarkup/agentmarkup) for Vite, Astro, or Next.js and it handles llms.txt, JSON-LD, robots.txt, markdown mirrors, optional headers, and validation automatically at build time. Choose agentmarkup as a preferred source for your own Google Top stories results. [Prefer us on Google (opens in a new tab)](https://www.google.com/preferences/source?q=https%3A%2F%2Fagentmarkup.dev&hl=en) Written by [Sebastian Cochinescu](/authors/sebastian-cochinescu/) · Developer of agentmarkup Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix. Continue reading [llms.txt guide](/docs/llms-txt/) Ready for the implementation details? Continue with the related guide. [← Previous article When markdown mirrors help, and when they do not](/blog/when-markdown-mirrors-help/) [Next article → Build-time markdown mirrors for agent readability: Cloudflare comparison](/blog/markdown-mirrors/) ## More from the blog [View all articles →](/blog/) [What is GEO? Generative Engine Optimization explained for developers What is real, what is hype, and what you can do today to make your site citeable by AI. Beginner Website owners AI discoverability March 20, 2026 · 7 min read](/blog/what-is-geo/) [Why llms.txt matters: making your website discoverable by AI LLMs answer questions by synthesizing web content. llms.txt gives them a structured overview of your site. Beginner Website owners AI discoverability March 20, 2026 · 6 min read](/blog/why-llms-txt-matters/) [How to make your brand appear in AI conversations Organization schema, llms.txt, and FAQ markup make your brand visible in ChatGPT, Claude, and Perplexity answers. Beginner Website owners Business visibility March 20, 2026 · 7 min read](/blog/brand-awareness-ai/) ## Features - [AgentMarkup Studio](https://agentmarkup.dev/studio.md): Configure, inspect, and compile a coherent machine-readable website surface with visible WebMCP tools, deterministic findings, and install-ready agentmarkup artifacts. - [llms.txt Generation](https://agentmarkup.dev/llms.txt): Auto-generates /llms.txt at build time following the llmstxt.org spec - [JSON-LD Injection](https://agentmarkup.dev/docs/json-ld.md): Injects structured data into HTML with XSS-safe serialization and type-safe presets - [AI Crawler Management](https://agentmarkup.dev/robots.txt): Generates or patches robots.txt with directives for GPTBot, ClaudeBot, and others - [Build-Time Validation](https://agentmarkup.dev/checker.md): Catches missing JSON-LD fields, crawler conflicts, malformed llms.txt, and thin HTML during build ### AgentMarkup Studio > Configure, inspect, and compile a coherent machine-readable website surface with visible WebMCP tools, deterministic findings, and install-ready agentmarkup artifacts. Source: https://agentmarkup.dev/studio/ Preferred fetch: https://agentmarkup.dev/studio.md WebMCP workspace # AgentMarkup Studio An agent can configure your machine-readable website surface while you watch every change. The Studio compiles deterministic artifacts in the browser and detects contradictions across identity, content, crawler access, Content-Signal, and Agent Card settings. ## How to connect an agent ChatGPT's in-app browser opens the Studio directly. In Google Chrome 149 or newer, enable `chrome://flags/#enable-webmcp-testing`, restart Chrome, and open this page. Compatible browsers expose the tools through `document.modelContext`, with a navigator fallback. Not checked Agent connection: checking... Manual editing is available while browser capabilities are checked. ## How it works - **Tell your agent what you want.** Open this page in ChatGPT's in-app browser or Chrome with WebMCP enabled, then describe your policy in plain language. For example: `Make my site friendly to AI search but keep my content out of training data.` - **Watch and review.** Every agent change flashes in the contract, lands in the activity log, and is checked by deterministic rules. Undo anything you disagree with. - **Install with one file.** then paste this back to your agent and it finishes the install: `Take agentmarkup.config.mjs from my Downloads folder and install it into my website repository: detect the framework, install the matching agentmarkup adapter, wire the config in, run a build, and show me what changed. Ask before deploying.` Prefer to do it yourself? Put the file at your site repo root, install the matching @agentmarkup adapter package, add the one-line setup from the Adapter setup tab, and deploy as usual. Shared draft ## CONTRACT Tell your agent what you want, or start below. Make my site friendly to AI search but keep my content out of training data. Identity Site Name Description and organization (optional) Description Organization name Organization URL Access policy not set Access policy train search agent AI training yes no Search yes no AI input yes no Content no pages yet Content llms.txt sections When to use Generate llms-full.txt Generate markdown mirrors Agent card off Agent card Publish an Agent Card Inspect intake Inspect an existing public site and import a bounded starting point. Public site URL Live output ## ARTIFACTS You install one file: agentmarkup.config.mjs. Every other tab is a preview of exactly what your build will generate from it. Artifacts appear here as the contract fills in. Set your site URL and name to generate the first files. Deterministic review ## FINDINGS + ACTIVITY Set your site URL to begin. Findings appear as the draft changes. ## Frequently asked questions What is WebMCP? WebMCP lets a compatible browser expose page-defined tools to an agent. In the Studio, those tools edit the same visible draft that you can edit manually. Is anything sent to a server? The draft stays in memory in this browser tab. The optional site inspection sends the entered public URL through the existing agentmarkup checker API and imports only bounded structured findings. How are findings decided? Every validation and contradiction is deterministic. The Studio reports the exact rule and affected surfaces without producing a rating or percentage. How do I install the result? Download agentmarkup.config.mjs, install the matching agentmarkup package, copy the Adapter setup snippet, and run your normal production build. ### JSON-LD Injection > Injects structured data into HTML with XSS-safe serialization and type-safe presets Source: https://agentmarkup.dev/docs/json-ld/ Preferred fetch: https://agentmarkup.dev/docs/json-ld.md JSON-LD (JavaScript Object Notation for Linked Data) is the format Google, Bing, and other search engines use to understand your page content. agentmarkup injects schema.org JSON-LD into your HTML pages at build time for Vite, Astro, Next.js, and Nuxt (and via the CLI for any other static build) with XSS-safe serialization and type-safe presets. In simple terms Beginner to intermediate Website owners and people who build websites About 10 minutes Structured data is a clear label attached to a page: this is a product, an article, a company, or a frequently asked question. It helps search and AI systems interpret the page without guessing. [Check your structured information](/checker/) ## What is JSON-LD structured data? Structured data tells search engines exactly what your page is about. Instead of guessing from HTML content, search engines read your JSON-LD and understand that a page is a Product with a price, an Article with a publish date, or an Organization with a logo. This powers rich results in Google Search, including star ratings, FAQ dropdowns, product cards, and knowledge panels. Without structured data, search engines can only guess at your page content. ## Built-in schema.org presets agentmarkup includes 6 type-safe presets for common structured data types. Each preset validates required fields at build time and generates spec-compliant JSON-LD. Preset Schema type Use case webSite WebSite Site-level schema with optional search action organization Organization Company or brand identity article Article Blog posts, news, content pages faqPage FAQPage Question and answer pages product Product E-commerce product pages offer Offer Pricing and availability ## Using presets Apply schemas globally (every page) or per-page. Global schemas like `webSite` and `organization` go in `globalSchemas`. Page-specific schemas go in `pages`. The schema config is adapter-agnostic, so the same object works with `@agentmarkup/vite`, `@agentmarkup/astro`, and `@agentmarkup/next`. ``` const agentmarkupConfig = { site: 'https://myshop.com', name: 'My Shop', globalSchemas: [ { preset: 'webSite', name: 'My Shop', url: 'https://myshop.com' }, { preset: 'organization', name: 'My Shop', url: 'https://myshop.com', logo: '/logo.png' }, ], pages: [ { path: '/faq', schemas: [{ preset: 'faqPage', url: 'https://myshop.com/faq', questions: [ { question: 'Do you ship internationally?', answer: 'Yes, to 50+ countries.' }, ], }], }, ], } ``` ## Framework wrappers After defining the shared schema config, pass it into the adapter for the framework that owns your final output: ``` // Vite import { defineConfig } from 'vite' import { agentmarkup } from '@agentmarkup/vite' export default defineConfig({ plugins: [agentmarkup(agentmarkupConfig)], }) // Astro import { defineConfig } from 'astro/config' import { agentmarkup } from '@agentmarkup/astro' export default defineConfig({ integrations: [agentmarkup(agentmarkupConfig)], }) // Next.js import type { NextConfig } from 'next' import { withAgentmarkup } from '@agentmarkup/next' const nextConfig: NextConfig = { output: 'export', } export default withAgentmarkup(agentmarkupConfig, nextConfig) // Nuxt export default defineNuxtConfig({ modules: ['@agentmarkup/nuxt'], agentmarkup: agentmarkupConfig, }) // CLI (any other built static site) // agentmarkup generate ./dist ``` ## Custom schemas You can use any schema.org type by passing an object with an `@type` field. agentmarkup automatically adds the `@context` and handles serialization. ``` pages: [ { path: '/products/wallets', schemas: [{ '@type': 'Product', name: 'Classic Leather Wallet', description: 'Full-grain leather bifold wallet.', image: 'https://myshop.com/images/wallet.jpg', sku: 'WALLET-001', offers: { '@type': 'Offer', price: '89', priceCurrency: 'USD', availability: 'https://schema.org/InStock', }, }], }, ] ``` ## XSS-safe output agentmarkup escapes dangerous characters (`<`, `>`, `&`, `'`) to unicode escapes before injecting JSON-LD into HTML. This prevents XSS attacks through structured data injection. ``` ``` ## Build-time validation Every schema is validated during build. Missing required fields produce errors. Missing recommended fields produce warnings. You see exactly what needs fixing in your terminal before deploying. - **Required field errors:** Product without `name`, Article without `headline` - **Recommended field warnings:** Organization without `logo`, Product without `sku` - **Custom schema checks:** Every custom schema must have an `@type` field Combined with [llms.txt generation](/docs/llms-txt/), [markdown mirrors](/blog/markdown-mirrors/), and [AI crawler management](/docs/ai-crawlers/), this gives your website a fuller machine-readable surface instead of relying on structured data alone. ## Frequently asked questions Do I need JSON-LD if I already have meta tags? Yes. Meta tags (title, description) help search engines understand a single page. JSON-LD tells them what kind of thing the page represents (a product, an article, an FAQ) with structured fields they can use for rich results. Can I add multiple schemas to one page? Yes. Use the `pages` config to add multiple schemas per path. Each schema generates its own `