The 5 files AI reads before your website

You've put real work into your website. The copy is sharper, the case studies are up, the services are clear. And yet when a buyer asks ChatGPT for a company like yours, you're not in the answer.

Often that's not because the content is wrong, but because the AI never got far enough to read it. Before a tool quotes a word of your site, it checks five things: your robots.txt, your sitemap, your structured data, your llms.txt and your headings. If one of them is missing or wrong, your best page can go unread.

None of these are new, and most sites have had the first two for years. What's changed is who's reading them. The good news is that each one takes a few minutes to check, so here's what each file does and how to check yours.

1. robots.txt: are AI tools allowed in?

robots.txt is a short text file at the root of your site that tells crawlers which pages they may read. If it blocks an AI search crawler, that tool can't find your pages for its answers, and nothing else on this list matters.

Each AI company uses named crawlers, and the names matter because you can allow or block them one by one. Most companies now run separate crawlers for search and for training. The ones worth knowing:

The split between search and training crawlers is useful. Some teams block training crawlers but still want to be found in AI answers, and that's a perfectly reasonable choice. OpenAI, for example, treats each setting separately, so blocking GPTBot doesn't remove you from ChatGPT search. Blocking the search crawlers is the one that keeps you out of answers.

There's a second place a block can hide. Since 1 July 2025, Cloudflare has blocked AI crawlers by default on new domains (The Register). If your site sits behind Cloudflare, your robots.txt can look perfectly open while Cloudflare turns the crawlers away before they arrive.

How to check: go to yoursite.com/robots.txt. Look for any of the search crawlers above followed by Disallow: /, or a blanket User-agent: * with Disallow: /. Then, if you use Cloudflare, check its bot settings for an AI crawler block. In Webflow, robots.txt lives under Site settings, then SEO.

2. Sitemap: what exists?

A sitemap is a list of every page you want found, usually at yoursite.com/sitemap.xml. Crawlers use it to discover pages they'd otherwise have to stumble across through links, and the last-updated date on each entry tells them what's changed.

The common problems are quiet ones. Old pages that redirect or no longer exist. Draft or test pages that were never meant to be public. A sitemap that isn't mentioned in robots.txt, so crawlers have to guess where it is.

How to check: open yoursite.com/sitemap.xml and scan it. Every URL should be a live page you'd be happy for a buyer to land on. Then check that your robots.txt has a line pointing to it, such as Sitemap: https://www.yoursite.com/sitemap.xml. Webflow builds the sitemap for you; pages set to exclude from search are left out of it.

3. Structured data: what is each page?

Structured data (often called schema) is a hidden block of code that labels a page in a format machines read without guessing. A person can tell a pricing page from a blog post at a glance. A crawler is much more confident when the page says so outright.

For a B2B site, a handful of types cover most of it:

• Organization on the homepage: your name, logo, website and social profiles
• Service on each service page: what you offer, and the price where you publish one
• Article on blog posts: the headline, author and dates
• FAQPage where a page answers common questions
• BreadcrumbList to show where a page sits on the site

It won't rescue a vague page, but it removes the guesswork from a clear one. It's also where facts like your price or location can be stated plainly, in a form that's easy to lift.

How to check: paste a page into Google's Rich Results Test or the Schema Markup Validator. Start with your homepage and your main service page. If nothing comes back, there's no structured data on that page.

4. llms.txt: the short version for AI tools

llms.txt is a plain-text file at yoursite.com/llms.txt that summarises who you are and links to your most important pages. It was proposed in 2024 as a simple way to hand AI tools the short version of a site, without the menus, scripts and layout in the way (llmstxt.org).

To be honest about where it stands: it's a proposal, not a standard, and not every AI tool reads it yet. It's on this list because it costs very little. Ten minutes of writing, and it's there for the tools that do check.

A useful one covers:

1. One or two sentences on what the company does and who it's for
2. Links to your key pages, each with a line on what it covers (services, pricing, case studies, contact)
3. Any facts you want stated correctly, such as your location and starting prices

How to check: go to yoursite.com/llms.txt. If it returns a 404, you don't have one. If it does exist, read it back and check that every link works and every fact is still true.

5. Headings: what does each section answer?

This is the one that's easiest to overlook, because it doesn't look like a technical file at all. AI tools rarely quote a whole page. They lift the passage that answers the question they were asked, and headings are how they find it.

Most B2B sites are written to be scrolled, not searched. Headings like "Built for growth" or "Our approach" read well in a design file, but they tell a crawler nothing about what sits underneath. Not because the writing is poor, but because it was written for a person who's already on the page and willing to read on.

The fix is small. Phrase headings the way a buyer would ask the question, and put the answer in the first line or two underneath. "How much does a Webflow build cost?" followed by a number gives an AI tool something it can quote. "Pricing that works for you" doesn't.

How to check: read only the headings on your main service page, top to bottom. If they sound like questions a buyer would ask, and each one is answered straight away, you're in good shape. If they read like slogans, that's where to start.

The five checks on one page

If you only have five minutes, open robots.txt. It's the one that can quietly undo everything else, and it's the quickest to fix.

None of this guarantees a mention. AI answers change from week to week, and what other sites say about you matters as much as what your own site says. But these five decide whether you're in the running at all, and they're the part that's fully in your hands.

Sources

• Overview of OpenAI crawlers (OpenAI)
• Does Anthropic crawl data from the web? (Anthropic)
• Cloudflare creates AI crawler tollbooth (The Register, 1 July 2025)
• The llms.txt proposal (llmstxt.org)
• Rich Results Test (Google)
• Schema Markup Validator (schema.org)