Technical SEO for AI crawlers: what changes and what does not
Rendering, robots directives, latency budgets and structured data — a practical checklist for making a site legible to both Googlebot and AI retrieval crawlers.
Most of the fundamentals carry over
AI retrieval crawlers are, mechanically, ordinary crawlers with tighter patience. Server-rendered HTML, clean canonicals, fast responses and honest status codes serve them exactly as they serve search engines.
If a page needs client-side JavaScript to reveal its main content, assume the retrieval crawler sees an empty shell and plan accordingly.
The parts that genuinely differ
Three areas need explicit decisions rather than inherited defaults.
- Robots policy: decide per user-agent whether AI crawlers may fetch, and document the commercial reasoning.
- Chunk boundaries: headings and section structure determine what gets retrieved as a unit.
- Freshness signals: visible and structured dates matter more when the consumer is summarising rather than ranking.
A latency budget you can defend
Set a server response budget of 200ms at the median and treat anything above 600ms as a defect with an owner. Retrieval crawlers time out quietly; you will not get an error report, only absence.
Pair that with a strict CLS discipline — reserved image dimensions, no late-injected banners — so the human experience and the machine experience improve from the same work.