Every so often you hear of content being indexed and exposed on Google and other search engines. Often the issue is not with the search engine but with the site that hosts the information.
Those sites often do not set the proper blocking mechanisms in place to communicate to Google and other search engines that the content should not be indexed and shown within the search results.
That is what happened recently with Anthropic's Claude Chats showing up in Google, Bing and other search engines.
Wired's story, "Private Claude Chats Exposed in Google and Bing Search Results," explained how private chats from Claude were found on Google, Bing and other engines — including politics, health discussions and other sensitive material.
"Claude allows users to share with other people 'snapshots' of chats by creating a public URL to a specific chatbot thread." — Wired
Wired added that "the reasons some of these URLs were indexed by major search engines comes down to the basic functions of websites, search engines, and the collision of the two when generative AI gets in the mix."
Over the weekend, a site:claude.ai/share search returned hundreds of Claude chats. Those results have since been removed.
The primary issue is that if you block using a robots.txt directive AND use noindex on that page, search engines like Google and Bing won't be able to see the noindex tag — because the crawlers can't access the page with the directive in place.
Glenn Gabe said on X he wished the journalists had spoken to an SEO before covering the story.
"It's filled with bad information. If you block via robots.txt AND noindex the page, Google and Bing cannot see the noindex tag since they can't crawl the page and see the tag in the HTML."
This is not new information. Google's own documentation carries a large notice at the top in bold with red highlights:
"Important: For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it."
Ned Adriance, a spokesperson from the Google Search team, told Wired:
"Neither Google nor any other search engine controls what pages are made public on the web, and these pages were indexed across many search engines."
"We give site owners clear controls to decide whether pages can be crawled or indexed, and we always respect those directives."
Microsoft Bing and Anthropic did not provide comment to Wired.
This shows the importance of consulting with an SEO who understands how to ensure the right pages are indexed — and the wrong ones aren't.
Never use robots.txt and noindex together. The single technical lesson here: pick one. To prevent indexing, allow the crawl so the noindex can be read.
Block genuinely private content with authentication, not meta tags. noindex is a request to search engines, not access control. For sensitive material, a structure where anyone with the URL can view it is the problem.
Set indexing policy when you design a "share link" feature. User-generated public URLs like Claude's shared snapshots are both a product feature and an SEO surface — a textbook case for involving SEO at the design stage.
Run site: queries against your own domain regularly. Checking site:domain/path for unintended indexing takes seconds — it is how this case surfaced.
Understand that removal is after-the-fact. The results were removed, but the Wired coverage and screenshots remain. De-indexing is not reputational recovery.
Classify it as technical debt. A conflict between crawl and index directives falls under "fix now" in the framework from Technical Debt in SEO — blocking indexation of important pages, or exposing pages that should be blocked.
Audit AI products' search surfaces separately. Public URLs generated by generative AI services often fall outside conventional site audit scope — on checking how AI exposes your business, see The AI Entity Footprint Audit.
Claude lets users create a public URL to share a snapshot of a chat thread, and those URLs were indexed. The issue was not the search engines but the blocking mechanisms on those pages not being configured correctly.
Blocking via robots.txt prevents crawlers from accessing the page, so they never see the noindex tag in the HTML. The noindex therefore doesn't take effect and the page can still surface if other pages link to it — as Google's documentation states in bold red at the top.
Google Search spokesperson Ned Adriance said neither Google nor any other search engine controls what pages are made public, that the pages were indexed across many engines, and that Google gives site owners clear crawl and index controls and always respects those directives.
With authentication, not meta tags. noindex is a request to search engines rather than access control, and any structure where knowing the URL grants access is unsuitable for sensitive information.
To apply what you just read to your own site, start with a free audit of where things are now.
A strategist replies within 24 hours on business days.
AI-era SEO is the Wild West again. After a decade inside Microsoft Bing, Duane Forrester says stop chasing the leading edge and work the chunky onions. What practitioners should drop, and what still holds.
The Google Search favicon bug is a week old and Google has confirmed it is on their end. Here is how to tell whether it is your site, and why this week's CTR drop should not enter your reporting as a result.
2027 SEO strategy shifts from ranking to being understood, trusted, and cited. Here are the priorities: AI answer engines, E-E-A-T, brand mentions, zero-click, and structured data.