How do I make a new website AI-ready before launch?

Why check AI readiness before launch, not after?

Because launches break things quietly. Two common failures:

• The staging block goes live. A test site is often told to keep all crawlers out with a Disallow: / rule in robots.txt, the file that tells crawlers which pages they may visit. If that rule ships, search and AI crawlers are shut out on day one, and nothing on the page looks wrong.
• Old addresses stop working. A redesign renames pages, and every link other sites built to the old addresses now hits an error page.

Both are cheap to catch before launch and slow to recover from after it. Google says a new site with few links pointing to it is exactly the kind that benefits from a sitemap, because crawlers find pages by following links from pages they already know. Google Search Central

What is the pre-launch AI-readiness checklist?

The sections below give the detail, with the source for each rule.

Which crawlers should robots.txt let in?

Google says robots.txt "tells search engine crawlers which URLs the crawler can access on your site", and warns that it "is not a mechanism for keeping a web page out of Google". To keep a page out, use a noindex tag or a password instead. Google Search Central

For ChatGPT, OpenAI separates three crawlers (OpenAI):

• OAI-SearchBot finds sites for ChatGPT search. "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."
• GPTBot collects content that may be used to train OpenAI's models. Blocking it is a training choice, separate from search.
• ChatGPT-User fetches pages when a person asks ChatGPT to; OpenAI says robots.txt rules may not apply to it.

OpenAI says robots.txt changes take about 24 hours to reach its systems, so fix a mistake before launch rather than after. For the full list of AI crawlers, see which AI crawlers to allow.

Do I need llms.txt and Markdown pages?

Not for Google. Google says there are no extra requirements to appear in AI Overviews or AI Mode, and "you don't need to create new machine readable files, AI text files, or markup to appear in these features." Google Search Central

llms.txt is a proposal, made by Jeremy Howard in September 2024, for a Markdown file that gives AI agents "brief background information, guidance, and links to detailed markdown files." It sits at the site root; the only required part is a top heading with the site's name. The same proposal suggests pages offer "a clean markdown version" at the same address with .md added. llmstxt.org

Our view: both are optional, cheap to add at launch, and easy for AI tools to read. We publish both on our own site. More in what llms.txt is and Markdown pages for AI crawlers.

What should the About page and structured data say?

The About page is where an engine checks who you are. Write it plainly: your name, what you do, where you do it, and links to your official profiles.

Put Organization structured data (code that states facts for machines) on the home page or About page. Google says it helps "disambiguate your organization in search results" and recommends properties such as url, logo, address, telephone and sameAs, which links to your pages on other sites. Google Search Central

Then match your Google Business Profile to the new site. Google says the name "should reflect your business's real-world name, as used consistently on your storefront, website, stationery, and as known to customers." Google Business Profile Help A launch that changes your phone number or address is the moment the two drift apart.

How should redirects work when URLs change?

Make a list of every old URL, then decide where each one should go. Google recommends "server side permanent redirects from the old URLs to the new URLs" and says to "keep the redirects for as long as possible, generally at least 1 year", so its systems can move all the signals over, including links on other sites. Google Search Central

Send each old page to its closest new match. A page about teeth whitening should land on the new teeth-whitening page, not the home page.

What should I check on launch day and in the week after?

1. Open robots.txt and the sitemap on the live domain, not staging.
2. Spot-check 20 old URLs and confirm each redirects once, to the right page.
3. Submit the sitemap in Google Search Console.
4. Run a scan that reads the site the way AI crawlers do.

For step 4, our free AI Access scan scores a site 0 to 100 across six areas: crawl and access, structured data, entity clarity (whether your name, address and phone agree everywhere), machine-readable files such as llms.txt, intent signals, and trend over time. It runs 30+ checks on every page it can find, and a re-scan confirms each fix worked. The score is free; full findings come with plans from $99 a month. Run it from the audit page.

Sources

1. Introduction to robots.txt — Google Search Central. Read Sep 24, 2026.
2. Overview of OpenAI crawlers — OpenAI. Read Sep 24, 2026.
3. Learn about sitemaps — Google Search Central. Read Sep 24, 2026.
4. Site moves with URL changes — Google Search Central. Read Sep 24, 2026.
5. The /llms.txt file — llmstxt.org. Read Sep 24, 2026.
6. AI features and your website — Google Search Central. Read Sep 24, 2026.
7. Organization structured data — Google Search Central. Read Sep 24, 2026.
8. Guidelines for representing your business on Google — Google Business Profile Help. Read Sep 24, 2026.