The five files that decide whether AI can read your site
When someone asks ChatGPT or an AI overview about your category, something has to decide whether your site is worth quoting. That decision does not happen in your hero section. It happens in a handful of small files most sites never think about. None of them are difficult, and all of them are easy to get wrong or leave out entirely.
Which five files decide whether AI can read a site?
- robots.txt. This tells crawlers, including the new AI ones, which parts of your site they are allowed to read. Get it wrong and you can accidentally block the very bots you want quoting you. It is worth checking you are not disallowing the crawlers that now matter.
- Your XML sitemap. A machine-readable map of every page worth indexing. Without it, a crawler has to guess at your structure by following links. With it, you hand over the full list. It is the difference between someone finding your work and someone stumbling across part of it.
- Structured data. This is where you tell an engine what a page means, not just what it says. That this is an article, written by this person, published on this date, about this topic. Answer engines lean on it heavily, because it removes the guesswork. A page without it is readable. A page with it is legible.
- llms.txt. The newest of the five. A plain-text file that points AI models at the content you most want them to use, in a form they can parse cleanly. Think of it as a reading list you hand the model, rather than leaving it to work out what matters on its own.
- Clean, structured headings. Not a file, but it belongs here. A page with one clear title and a sensible heading order is far easier for a model to summarise and quote than a wall of styled text. Structure is content, as far as a machine is concerned.
How much work is it to get them right?
None of this replaces good writing or a clear offer. It just makes sure the good writing can be read by the things now deciding who gets cited. If you have never checked these five, that is the fastest, least glamorous visibility win available to you, and most of it is an afternoon's work.
FAQs
What is llms.txt?
What is llms.txt?
llms.txt is a proposed standard: a plain-text file at the root of your site that points AI models at your most important pages, with a short description of each. Support across the big assistants is still uneven, but it takes an hour to write and gives any model that reads it a clean map of your site.
Which AI crawlers should robots.txt allow?
Which AI crawlers should robots.txt allow?
If you want to be cited, allow the bots that power AI search and answers, including OAI-SearchBot and ChatGPT-User from OpenAI, Claude-SearchBot and Claude-User from Anthropic, PerplexityBot and Googlebot. Training bots such as GPTBot, ClaudeBot and Google-Extended can be allowed or blocked separately, depending on how you feel about your content being used for training.
Does Webflow create a sitemap and robots.txt automatically?
Does Webflow create a sitemap and robots.txt automatically?
Yes. Webflow generates an XML sitemap automatically and lets you edit robots.txt in the site settings. Structured data and llms.txt need adding yourself, through custom code or a hosted file, and both take an afternoon at most.
What structured data should a B2B website have?
What structured data should a B2B website have?
Organization markup on the homepage, Article or BlogPosting on articles with author and date, FAQPage on genuine question-and-answer sections, BreadcrumbList for navigation and Service or Product on offer pages. Keep the markup consistent with what is visible on the page, or engines learn to ignore it.
How do we check whether our site is readable by AI?
How do we check whether our site is readable by AI?
Open robots.txt and check you are not blocking answer-engine bots, confirm the sitemap lists every page that matters, and run key pages through Google's Rich Results Test. Then view the page source and make sure the important text is in the HTML itself. If all four pass, the basics are in place.