Back to blog
AI SEO

JavaScript Links Are Invisible to AI Search — Results From a 41-Day Experiment

JavaScript Links Are Invisible to AI Search — Results From a 41-Day Experiment

"Google renders JavaScript" has been the standard answer to client-side rendering concerns for years, and it got easier to believe in 2019 when Googlebot went evergreen on the latest Chromium.

The increasing flow of search traffic through AI assistants renders that wisdom incomplete, because the crawlers for AI assistants behave nothing like Googlebot.

To get numbers rather than assumptions, a controlled experiment set up one site with half its internal links hard-coded in HTML and the other half injected by JavaScript, with a logging layer recording every bot request for 41 days.

The results were more lopsided than expected. Google's crawler stack was the only one that executed JavaScript and followed the injected links. Almost none of that work came from Googlebot, the crawler that actually builds the search index — Googlebot reached only 2% of the JavaScript-linked pages. Meanwhile GPTBot, ClaudeBot, Bingbot, PerplexityBot, Meta's and Amazon's crawlers, and every other tracked bot found exactly zero.

Experiment Design

The test site is a Brazilian business classification directory of roughly 2,400 pages. The experiment ran on the site's hierarchy: 21 top-level sections branching into divisions, groups, and classes, 1,062 pages total. The remaining flat leaf pages sat outside the test.

The 21 sections were split into two groups, alternating to avoid clustering bias:

  • HTML group (11 sections): every child link hard-coded in raw HTML so any bot can parse and follow them.
  • JS group (10 sections): every child link, at every level, injected by JavaScript after page load. The raw HTML contained an empty div — a bot that does not execute JavaScript sees no links at all.

Every alternative discovery path was removed:

  • Sitemaps: disabled so all sitemap URLs returned 404
  • Breadcrumbs: removed so deep pages could not leak links back up
  • Internal hierarchy panels: removed for the same reason

The only way to reach a division page in the JS group was to render JavaScript on its parent section page.

Middleware logged every bot request — path, user agent, referrer, IP, timestamp. Googlebot and Bingbot IPs were verified via reverse DNS to filter spoofed user agents.

One measurement detail matters. Section pages for both groups were always linked from the index in plain HTML. A bot hitting a JS-group section page proves nothing. The experiment signal starts one level deeper: a hit on a JS-group division page is only possible if the bot rendered the parent's JavaScript and followed the injected link.

Phase 1: The Initial Crawl Burst

Crawlers arrived fast and crawled hard for the first 72 hours.

GPTBot swept the site within hours of deployment and crawled all 759 pages of the HTML group on day one, down to class level. In the JS group it hit all 10 section pages and stopped. It never reached a single division page.

ClaudeBot followed the same pattern a day later: deep, systematic HTML crawling and a hard stop at the JavaScript boundary.

Google's two crawlers behaved differently from each other. GoogleOther reached its first JS-injected division page 54 minutes after launch. Googlebot took 9.9 hours. Both worked progressively deeper into the JS hierarchy over following days, which is only possible if they rendered the parents and followed the injected links.

The burst lasted three days. On day four, discovery dropped about 90% for both groups and coverage froze.

Only Google Executed the JavaScript

The scorecard at the end of Phase 1, after 27 days, covering hierarchy pages only (division level and deeper), where a JS-group hit is unambiguous proof of JavaScript execution. HTML sections hold 748 hierarchy pages, JS sections 293.

CrawlerRenders JS?HTML hierarchy pagesJS hierarchy pages
GooglebotYes35 (5%)7 (2%)
GoogleOtherYes495 (66%)142 (48%)
GPTBot (OpenAI)No7480
ClaudeBot (Anthropic)No7480
BingbotNo2823
OAI-SearchBot (ChatGPT search)No100
Meta-ExternalAgent (Meta AI)No5370
AmazonbotNo7350
ChatGPT-User (live browsing)No460
PerplexityBotNo00

Even Google Barely Touched the JavaScript Pages

The two Google rows do very different jobs, and the difference matters more than the raw counts.

GoogleOther passed the rendering test. Googlebot — the crawler that feeds Search — mostly sat it out.

So the 66% and 48% on the GoogleOther row say nothing about whether those pages can appear in Google. The row that speaks to search visibility is Googlebot: 5% of the HTML group and 2% of the JS group. Six of the 10 JS sections got no Googlebot hierarchy crawl at all in 27 days.

One caveat: GoogleOther out-crawled Googlebot roughly 14:1 on HTML pages here. That reads as a crawl-budget artifact of a brand-new domain with no external links, not evidence that GoogleOther is the more thorough crawler. On an established site, Googlebot's share would likely be much higher.

What the split does show is that a site owner watching raw crawl logs can badly overestimate search coverage by counting every Google user agent as one crawler.

JavaScript Was Discovered Later

GoogleOther's JS-linked pages were discovered 18% to 27% later than its HTML-linked pages at every depth. The lag compounds: Google must fetch the parent, queue it for rendering, execute the script, extract the URLs, then schedule the children. HTML links skip the middle steps.

Distribution was uneven. One JS section got 52 pages crawled while two others sat at one page each for weeks. Rendering appears to be a queue with its own budget, and some branches never came up.

One result ran against expectations: once GoogleOther discovered a JS-linked page, it recrawled it as fast as comparable HTML pages, and at two of three depth levels 24% to 27% faster. The JavaScript delay is entirely front-loaded in discovery. Once past that stage, a page is treated normally, sometimes better.

The pages that never get past it are the ones that hurt. They aren't ranking slowly. They just aren't there at all.

Every Other Crawler Ignored the JS Group

The middleware initially lumped Meta-ExternalAgent and Amazonbot into a generic "unknown bot" bucket. Splitting by user agent revealed them as two of the heaviest crawlers on the site, roughly 2,000 hits each. Meta-ExternalAgent gathers training data for Meta's AI models; Amazonbot feeds Amazon's search and AI products. Both crawled the HTML group deeply, hit the 10 JS-group section pages, and never went one level further.

Bingbot was active on 20 of Phase 1's 27 days, recrawled the 10 JS-group section pages on 18 of them, and never followed one injected link. Microsoft has documented an evergreen, Edge-based Bingbot capable of rendering JavaScript since 2019, but across 20 days of active crawling that capacity was not applied here.

PerplexityBot stayed on the homepage and robots.txt for all of Phase 1.

Real User Requests Were Blocked Too

ChatGPT-User deserves attention because it represents real user demand — the agent that fetches pages when someone asks ChatGPT a question.

During the experiment, real user queries pulled it to dozens of HTML-linked pages. Pages behind JavaScript links were unreachable for those same queries. The pages a real person was actively trying to reach were exactly the ones it couldn't get to.

Phase 2: Does Fixing It Work?

After 27 days, all JavaScript links were converted to hard-coded HTML. Nothing about the pages changed — same URLs, same content, same server. Only the links moved.

Fast Responders

GPTBot responded within two days, sweeping every formerly blocked section in under three hours and picking up 250 pages it had never crawled, against zero in 27 days of Phase 1.

Bingbot followed with a systematic sweep adding 212 pages across sections it had ignored for weeks, reaching class level where it had never passed the entrance.

Meta-ExternalAgent joined by day 41 with a sweep of its own.

Non-Responders

Amazonbot had finished its crawl and left on day 21, six days before conversion, so its JS-group coverage ended at zero.

ClaudeBot stayed at zero differently. It kept recrawling after the fix — hundreds of hits through day 41 — but only on pages it already knew. It never picked up a single newly linked page, apparently working from the same internal map it drew on the first pass.

The slowest responder, ironically, was Google. Its aggressive first-visit budget was already spent. By day 41, Googlebot had visited exactly one page in the converted sections and GoogleOther none.

Fixing link architecture after launch recovered AI crawler coverage quickly, but with Google, that first crawl of a brand-new site looks much harder to win back a second time.

What to Tell a Client

This is one site, one niche, one 41-day window, and a fresh domain with no external links pointing at the buried pages. Established sites with strong signals may see Googlebot invest more in rendering. The non-Google results leave less room for interpretation: eight crawlers failed to execute the JavaScript and follow the injected links across every crawl burst and recrawl cycle.

Audit what your raw HTML actually contains. Disable JavaScript and click through critical paths, or compare raw source against the rendered DOM. If navigation to revenue pages only exists after hydration, every AI crawler is blind to it.

Treat "Google renders JS" as the exception, and a partial one. Rendering happened, but the crawler that feeds Search reached 2% of the JS-linked pages. Check which Google user agent your logs are showing you before counting that coverage as visibility.

Prioritize server-side rendering for links above content. A widget that renders in JavaScript makes a page slightly worse. Navigation that renders in JavaScript can drop whole sections of your site out of every index feeding an AI assistant.

Fix it sooner rather than later. GPTBot re-swept within 48 hours of conversion and Bingbot within a week. Google took weeks and still had not returned by day 41.

Related Reading

This experiment shows why the technical accessibility and structured data items in the 2027 SEO priorities are prerequisites rather than options. It is also why phase one of the three-phase process for earning AI citations — making pages accessible to LLMs — gates the other two.

Bottom Line

The rendering question used to be whether Google could see your content. With AI assistants becoming a real referral channel and the web reshaping itself around agents, the question is now whether other crawlers can.

If your links live in JavaScript, they can't.

Frequently Asked Questions

Do AI crawlers execute JavaScript?

In this 41-day experiment, eight tracked non-Google crawlers — GPTBot, ClaudeBot, Bingbot, PerplexityBot, Meta-ExternalAgent, Amazonbot and others — followed exactly zero JavaScript-injected links.

Doesn't Google render JavaScript?

It does, but nearly all of that work came from GoogleOther. Googlebot, the crawler that builds the search index, reached only 2% of JS-linked pages and 5% of HTML-linked pages.

Does converting links to HTML fix it?

Partially. GPTBot re-swept within 48 hours and Bingbot within a week, but ClaudeBot picked up zero newly linked pages through day 41, and Google barely returned to the converted sections.

What should be checked first?

Disable JavaScript and click through critical paths, or compare raw source against the rendered DOM. If navigation to revenue pages exists only after hydration, AI crawlers cannot see it.

Where does your own site stand?

To apply what you just read to your own site, start with a free audit of where things are now.

A strategist replies within 24 hours on business days.

Read next