How fast does your site need to be for AI crawlers?

You've optimized your content. Your site has expertise, authority, and trust signals. But when an AI crawler reaches your page and faces a timeout—when your server takes 6 seconds to respond, or when LCP isn't done rendering for 5 seconds—the crawler moves on. Your content never gets indexed. Your citations never happen.

Speed kills visibility. Not in a slow-burn way. In an instant, hard-stop way.

AI crawlers operate under constraints that traditional search engines do not. They have strict timeout windows, limited compute budgets, and shallow patience. A page that ranks perfectly for Google can be completely invisible to ChatGPT, Perplexity, or Claude because it fails to load within the crawler's deadline. This chapter covers how Core Web Vitals work for AI, which metrics matter most, and what changes you need to make.

What Core Web Vitals measure and why they exist

Core Web Vitals are three metrics Google created to measure real user experience. They capture the moment the browser starts showing content, the moment users can interact with the page, and how stable the layout stays during load. Google treats them as ranking factors. But for AI crawlers, they are survival factors.

The three metrics are:

Largest Contentful Paint (LCP): How long until the biggest visual element loads. The target is under 2.5 seconds. This is what users see first—if your hero image, headline, or main content section takes 4 seconds to appear, both humans and crawlers have to wait.

Interaction to Next Paint (INP): How long it takes for the browser to respond after a user clicks something. The target is under 200 milliseconds. This metric tests whether your JavaScript is bloated or if your page freezes when crawlers simulate interaction.

Cumulative Layout Shift (CLS): How much the page bounces around while loading. The target is under 0.1. When elements shift unexpectedly, crawlers extract scrambled or incomplete content. Your carefully written paragraphs become garbled in the crawler's parsing.

For traditional search, these are hints. For AI crawlers, they are hard limits.

Why AI crawlers have different speed requirements than Google

Google's crawler distributes the work. It crawls millions of URLs across thousands of servers. It can afford to wait 10 seconds for a page to load because the wait is spread across an enormous infrastructure.

AI crawlers operate on per-query inference. When a user asks ChatGPT a question, Perplexity has 3 to 5 seconds to find, retrieve, and parse your content before the response needs to start generating. Every second matters. Every millisecond is counted against the crawl budget.

Studies of AI crawler behavior show that sites with First Contentful Paint under 0.4 seconds average 6.7 citations, while sites loading over 1.13 seconds average 2.1 citations. This is not correlation. This is causation. Slow pages get skipped.

ChatGPT's retrieval crawler has a documented timeout threshold. Pages that don't respond within that window are marked as unreachable and deprioritized in future crawl attempts. Perplexity's bot does the same. A site that takes 6 seconds to respond doesn't get a second chance.

Time to First Byte (TTFB): the metric AI crawlers care about most

TTFB measures the moment between sending a request and receiving the first byte back from the server. It is the purest measure of server responsiveness.

For AI crawlers, TTFB determines whether they even attempt the render. If your server takes 700 milliseconds to respond, the crawler has wasted 700 milliseconds of its 3 to 5 second window. The crawler abandons rendering entirely and falls back to parsing HTML only—which means your JavaScript-rendered content becomes invisible.

The benchmarks are clear. A TTFB under 200 milliseconds is considered exceptional. 200 to 500 milliseconds is acceptable but not ideal. Above 600 milliseconds, you have a server infrastructure problem.

Most performance articles treat TTFB as secondary to visual metrics. For AI crawlers, it is the gatekeeper. If TTFB fails, nothing else matters.

Common causes of slow TTFB include slow database queries behind every page load, overloaded hosting with too many sites per server, poor geographic routing of requests, and lack of content delivery network (CDN) caching. These are not design problems. These are infrastructure problems.

Largest Contentful Paint (LCP): waiting for the big asset

LCP measures the time until the largest visual element fully loads. This is usually a hero image, a background image, or a large text block. For content-heavy pages, it is the main content area.

An LCP under 2.5 seconds is the "good" threshold. AI crawlers cite pages in this range 2.8x more often than pages with slower LCP. The difference is not subtle.

The crawler does not wait for every asset to finish loading. But it does wait for the main content to appear. If your hero image is 4 megabytes and you have not optimized it for web, the crawler sits idle while the image downloads. By the time the image finishes loading, you are already over the crawler's timeout.

Common causes of slow LCP include oversized images without compression, unoptimized video embeds that block rendering, web fonts that are not subset and do not load fast, and third-party scripts that render above the fold. Each of these adds seconds to LCP.

The fix is not to wait longer. The fix is to make LCP happen faster. This means image optimization, lazy loading of below-the-fold content, asynchronous script loading, and potentially swapping out heavy third-party embeds for static HTML alternatives.

Interaction to Next Paint (INP): JavaScript execution performance

INP measures responsiveness. When a user (or a crawler simulating a user) interacts with the page, how long does it take for something to change on screen? The target is under 200 milliseconds.

For AI crawlers, INP reveals whether your JavaScript is optimized or bloated. A poor INP score signals that your page has too much code running on the main thread, or that code is not properly chunked and deferred.

Crawlers that simulate interaction do this to extract dynamic content. If your page loads static HTML but depends on JavaScript to render forms, conditional content, or interactive elements, the crawler needs to execute that JavaScript. If the JavaScript is slow, the crawler gets a poor INP score and marks your page as inefficient.

The hidden problem with client-side rendering is that most AI crawlers don't render JavaScript at all. Research shows that static HTML with schema markup achieves 94% parsing success, while JavaScript-rendered content lands at 23%. This means that if your content is hidden behind JavaScript, AI crawlers may never see it.

If your site uses client-side rendering, the INP problem becomes a visibility problem. The crawler cannot wait for JavaScript to execute, so it skips JavaScript-dependent content.

Cumulative Layout Shift (CLS): stability during extraction

CLS measures visual instability. When elements on the page move around during load, the Cumulative Layout Shift increases. The target is under 0.1.

For crawlers, CLS is not about user experience. It is about parsing accuracy. When the DOM shifts while the crawler is extracting content, it may grab the wrong elements. A headline may shift into another headline's space. A sidebar may push the main content out of position. The crawler extracts whatever text is at a position, not whatever text logically belongs there.

Common causes of high CLS include ad spaces that load late and push content down, images without specified dimensions that load and shift the layout, web fonts that swap and change spacing, and carousels that auto-play and shift content. Each causes the layout to move.

The fix is to reserve space for everything that will load. Specify image dimensions in HTML. Define font-display as swap so fonts don't block rendering. Load ads and dynamic content in reserved containers that do not cause layout shifts.

How to optimize for AI crawler speed

Speed optimization for AI crawlers is different from speed optimization for humans. Users care about when they see the page and when they can interact with it. Crawlers care about when the server responds and whether all content finishes rendering.

Start with TTFB. If your server takes more than 500 milliseconds to respond, nothing else matters. Audit your database queries, move to a faster hosting provider, implement caching at the application level, or use a CDN. The investment in TTFB pays off immediately in crawler coverage.

Optimize images. Every image on your page should be compressed, resized for its display dimensions, and served in modern formats like WebP with PNG fallbacks. Use a tool like TinyPNG or ImageOptim. For hero images, use a CDN with automatic optimization.

Defer non-critical JavaScript. Only load JavaScript that is absolutely necessary for the crawler to see content. Scripts that track analytics, load ads, or power interactive features should be deferred or loaded asynchronously. This keeps the main thread free and execution fast.

Server-side render critical content. If your site uses a client-side framework like React or Vue, render the critical path on the server and hydrate on the client. This ensures crawlers see content immediately. Static site generation (SSG) is even better—build your pages to HTML at deploy time.

Avoid large third-party embeds. If you embed YouTube players, Spotify players, or large external widgets, they block rendering. Replace them with placeholders that lazy-load on click, or extract the essential information into static HTML.

Reserve space for dynamic content. Define explicit dimensions for images, ads, and dynamic sections. This prevents layout shift and ensures the crawler extracts the right content.

Use a CDN. A content delivery network routes requests to the geographically closest server, reducing latency and TTFB. Most CDNs also offer automatic image optimization and compression.

What WEMASY includes for speed and crawler optimization

WEMASY's website builder includes hosting with automatic CDN caching, image optimization on upload, and server-side rendering of all site content. Pages built with WEMASY meet Core Web Vitals standards by default.

When you publish a site with WEMASY, your TTFB is optimized through our CDN, your images are automatically resized and compressed, and your HTML is served as static pages—not generated on demand. This means AI crawlers see your content immediately, without waiting for rendering or processing.

Learn what's included in WEMASY pricing plans.

Frequently asked questions