There's a category of visitor showing up on websites now that's neither a human clicking around nor a search engine quietly indexing pages. It's software acting on someone's behalf, trying to read a page, understand what it offers, and sometimes act on it directly. Most websites weren't built with it in mind.
Google Now Publishes Guidance for This, Specifically
Google's own web.dev site includes guidance titled "Build agent-friendly websites," treating AI agents as a distinct audience a site needs to design for, alongside human visitors and traditional crawlers. The specifics are concrete: use real semantic elements like <button> and <a> instead of a styled <div> pretending to be one, keep layouts stable rather than shifting unpredictably, link form labels to their inputs properly, write descriptive link and button text instead of "click here," and include alt text on images. Google's own stated philosophy behind the guidance is worth repeating directly: everything that makes a site agent-ready also makes it better for humans. This isn't a parallel, competing standard to good web development, it's the same one, applied more strictly.
Most AI Crawlers Don't Execute Your JavaScript
This is the detail with the most immediate, practical consequence. An independent analysis of more than 500 million GPTBot fetches found zero evidence of JavaScript execution. When GPTBot requests a page built as a React, Vue or Angular application, it downloads whatever HTML the server returns and moves on, it doesn't wait for the app to mount or for API calls to resolve. If the actual content only appears after client-side rendering finishes, GPTBot never sees it, regardless of how clean the underlying code is. The one real exception is Google-Extended, the crawler behind Gemini, which does render JavaScript because it inherits Googlebot's own infrastructure, a genuinely different behavior from the other major AI crawlers.
The Scale Is Already Real, Not Theoretical
GPTBot, Anthropic's Claude crawler, AppleBot and PerplexityBot combined account for roughly 1.3 billion fetches, about 28% of Googlebot's own crawl volume. In a single month, GPTBot alone made around 569 million fetches and Claude's crawler around 370 million. This isn't a speculative future traffic source. It's already a meaningful share of who, or what, is actually reading websites today, and it's the layer feeding what AI systems say about a business when someone asks.
What Actually Makes a Site Agent-Ready
- Server-side rendering or static generation, so the real content is present in the initial HTML response, not something that only appears after JavaScript runs.
- Semantic HTML elements (
<button>,<a>, proper heading hierarchy) instead of generic, styled containers standing in for them. - Form labels correctly linked to their inputs, so an agent can identify what information a field is actually asking for.
- Descriptive link and button text that states the actual action or destination, not "click here" or "learn more" with no context.
- Stable page layouts that don't shift unpredictably as content loads.
- Alt text on meaningful images, the same accessibility discipline covered in our Website Health Scorecard, doing double duty here.
Want to know if your site's technical foundation is actually agent-ready?
Run a Free SEO Health Check →Frequently Asked Questions
What is an AI agent in the context of website SEO?
An AI agent is software acting on a person's behalf, browsing a website, reading its content, and in some cases attempting to complete a task like filling a form or comparing options, rather than a human clicking through manually or a traditional search engine simply indexing pages. Google's own guidance now treats agents as a distinct audience a website needs to design for, alongside human visitors and search crawlers.
Do AI crawlers like GPTBot execute JavaScript?
No, based on an independent analysis of more than 500 million GPTBot fetches, which found zero evidence of JavaScript execution. GPTBot requests a page, reads whatever HTML the server returns immediately, and moves on, it doesn't wait for a React, Vue or Angular app to render. A site relying entirely on client-side rendering is effectively invisible to it. The one notable exception is Google-Extended, Gemini's crawler, which does render JavaScript because it inherits Googlebot's own infrastructure.
What does Google's agent-ready website guidance actually recommend?
Google's own web.dev guidance, "Build agent-friendly websites," recommends semantic HTML elements like <button> and <a> instead of styled <div>s, stable layouts that don't shift unpredictably, form labels properly linked to their inputs, descriptive link and button text, and alt text on images. Its own stated philosophy: everything that makes a site agent-ready also makes it better for humans.
How much traffic do AI agent crawlers actually generate?
A meaningful and fast-growing amount, not a rounding error. GPTBot, Claude's crawler, AppleBot and PerplexityBot combined account for roughly 1.3 billion fetches, about 28% of Googlebot's own crawl volume, with GPTBot alone making around 569 million fetches and Claude's crawler around 370 million in a single month.
Is server-side rendering necessary for AI agent visibility?
For the majority of current AI crawlers, effectively yes. Since GPTBot and similar crawlers don't execute JavaScript, content that only appears after client-side rendering completes is invisible to them regardless of how well-structured the underlying code is. Server-side rendering or static generation ensures the actual content is present in the initial HTML response these crawlers read.
Related reading: How to Rank in AI Search Results