AI Search Technical Checklist: Structured Data, llms.txt, robots.txt and Crawlable Content

By Ian Carlos Fajardo (Co-founder, Development Lead, BBZ Studio). Published . 6 minute read.

An AI search technical checklist covers four things: allow AI crawlers in robots.txt, make sure your important content is real HTML text that loads without heavy scripts, add structured data that labels your business facts, and optionally publish an llms.txt file that summarizes your site. Get these right and ChatGPT, Claude, Perplexity and Google can read your site the way you intend.

This is the developer side of AI visibility. Good content matters most, but it only counts if machines can reach it. Below is the checklist we run on every site we build at BBZ Studio, written so a business owner can follow along and a developer can act on it.

1. Is your robots.txt letting AI crawlers in?

Your robots.txt file lives at yourdomain.com/robots.txt and tells automated visitors which parts of your site they may crawl. AI companies run their own crawlers, and each has a name (called a user agent). If your file blocks them, those tools may not be able to read or cite your pages.

Common AI user agents include GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot, Claude-SearchBot and Claude-User from Anthropic, and PerplexityBot from Perplexity. Google uses a separate token, Google-Extended, to control whether your content is used for its AI models, while normal Googlebot rules still govern search.

  • Open your robots.txt and look for Disallow rules under AI user agents
  • Check for a blanket rule like User-agent: * with Disallow: / left over from a staging site
  • Check whether a security plugin, firewall or CDN setting is blocking bots at the server level
  • Make sure your robots.txt points to your XML sitemap

Pro tip: Crawler names change and new ones appear. Check each AI company's own documentation for the current list before you edit your rules.

2. Can machines read your content without running JavaScript?

Many modern websites build the page in the browser with JavaScript. Google can usually handle that, but not every crawler renders JavaScript fully, and some only read the raw HTML your server sends. If your headings, services and prices only appear after scripts run, some AI tools may see a nearly blank page.

The fix depends on your platform. WordPress and Shopify usually send real HTML by default. For custom React or similar apps, use server-side rendering, static generation, or prerendered fallback HTML for key pages. On our own site, we generate crawler-friendly HTML for each page at build time so the text is there from the first byte.

  • View the page source (not the inspector) and check your main text is there
  • Avoid putting key messages only in images, sliders or videos
  • Use real headings (H1, H2, H3) in a logical order
  • Give images descriptive alt text
  • Keep important content out of tabs or accordions that load on click from the server

3. Which structured data should you add?

Structured data is code, usually in JSON-LD format, that describes your page in a vocabulary machines understand (schema.org). It does not change how the page looks. It labels the facts so search engines and AI systems do not have to guess.

  • Organization or LocalBusiness: name, logo, contact details, address or service area, social profiles
  • WebSite: your site name and URL
  • Service and Offer: what you sell and, where you can, the price
  • FAQPage: the questions and answers on the page
  • BlogPosting or Article: author, date and headline for articles
  • BreadcrumbList: where the page sits in your site

Only mark up what is visible on the page and true. Validate everything with Google's Rich Results Test or the Schema Markup Validator before you ship.

4. Do you need an llms.txt file?

llms.txt is a proposed convention: a plain-text Markdown file at yourdomain.com/llms.txt that gives AI tools a short summary of your site and links to its most important pages. It is not an official standard and not every AI company says it uses it, so think of it as cheap insurance rather than a ranking factor.

A good llms.txt is short and factual. Include your business name, a one-paragraph description, your main services and prices, where you work, and links to your key pages. Keep it in sync with your website, ideally generated automatically from the same data, so it never goes out of date.

5. Speed, sitemaps and the basics that still matter

Crawlers have limited time per site. Slow servers, huge images and long redirect chains mean fewer pages get read. The same fixes that help Core Web Vitals help AI crawlers too.

  • An up-to-date XML sitemap listing every page you want found
  • Clean URLs with one canonical version of each page
  • HTTPS everywhere and no broken internal links
  • Compressed, correctly sized images
  • A fast host and caching

6. Are your business facts consistent everywhere?

This one is not code, but it is technical in its own way. AI assistants cross-check facts. Your business name, services, contact details and prices should match across your website, structured data, llms.txt, Google Business Profile and social profiles. When they disagree, assistants hedge or leave you out.

Pro tip: Keep one source of truth for your business facts, such as a simple data file or settings page, and generate your structured data, llms.txt and page copy from it. That is how we keep prices and services in sync across our own site.

The quick AI search checklist

  • AI crawlers allowed in robots.txt, sitemap linked
  • Main text present in the raw HTML
  • Logical headings and descriptive alt text
  • Organization, Service, FAQPage and Article structured data, validated
  • llms.txt published and kept in sync
  • Fast pages, HTTPS, clean canonical URLs
  • Consistent business facts across every source

If you would rather have someone check this for you, our free website audit covers these points. Every site we build at BBZ Studio, from the 1-page landing website to custom builds, ships with this setup as standard.

Frequently asked questions

Should I block AI crawlers from my website?

For most small businesses, no. Blocking AI crawlers can stop assistants like ChatGPT, Claude and Perplexity from reading and recommending you. Some publishers block training crawlers to protect their content, but businesses that want to be found usually benefit from allowing them.

Does llms.txt help with SEO?

Not directly. llms.txt is a proposed convention for AI tools, not a Google ranking factor. It is quick to add and can help AI tools understand your site, but structured data, crawlable content and good pages matter far more.

What's the best structured data for a small business website?

Start with Organization or LocalBusiness, then add Service or Offer for what you sell, FAQPage for your FAQ sections, and Article or BlogPosting for blog posts. Only mark up information that is visible on the page.

How do I know if my website is crawlable by AI?

Check your robots.txt for blocked AI user agents, view your page source to confirm the main text is in the HTML, and test your structured data with a validator. You can also ask an AI assistant about a specific page and see whether it can describe it.

Get a free website audit

Want to know how your website measures up? Request a free website audit or see our fixed-price bundles, starting at $497.