SEO

How Often Sites Block AI Crawlers: 14,318 Sites Measured

A Seonix study published on November 3 measured the robots.txt files of 14,318 sites and found a sharp split: sites block AI training bots far more often than the bots that fetch pages for AI answers. Only 12.2% of sites block a training bot by name, and just 2.3% block a search or answer bot.

How the study was done

Seonix started from 16,399 sites seen in the top 10 results of commercial searches, removed 46 of its own, and set 133 large platforms such as social networks and marketplaces aside into a separate group. That left 16,220 sites, and on November 3, 2026 the researchers sent one request for /robots.txt and one for /llms.txt to each and opened no other page. A readable robots.txt file came back from 14,318 of them, or 88.3% of the requests. The rest had no file, returned a web page, refused the request, or failed, and those sites sit outside every blocking share. Each bot was sorted by the job its operator says it does: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent and Amazonbot count as training and data collection, while OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User count as search and answers.

Training bots are blocked 5.3 times as often

The headline finding is the gap between the two kinds of bot. Training bots are blocked 5.3 times as often by name as search and answer bots: 1,744 sites name a training bot in a full block, against 330 that name a search or answer bot. Counting blocks through User-agent: * as well, the shares become 12.8% for training bots and 2.9% for search and answer bots. The split holds within one company: GPTBot, OpenAI's training crawler, is blocked on 8.0% of sites, while OAI-SearchBot, which fetches pages for ChatGPT answers, is blocked on just 0.7%. ClaudeBot sits at 7.4% against 0.5% for Claude-SearchBot. The most-blocked bot of all is ByteDance's Bytespider at 9.4%, followed by Common Crawl's CCBot at 8.8%, GPTBot at 8.0%, ClaudeBot at 7.4% and Google-Extended at 6.4%. On the answers side, ChatGPT-User and PerplexityBot are each blocked on 1.4% of sites, OAI-SearchBot and Claude-User on 0.7%, and Perplexity-User on 0.8%.

One template drives a large share of the blocks

A big part of the training-bot blocking comes from a single default rather than a deliberate decision per site. Seonix found that 35.8% of named training blocks sit in a file carrying Cloudflare's managed block, the template Cloudflare documents. Strip those files out and the shares fall to 8.2% for training bots and 2.1% for search and answer bots. Most sites that block training bots are careful to leave AI search alone: 1,414 sites, or 9.9% of the sample, block training bots by name while leaving every search and answer bot open, and fewer than 30 do the reverse. Large platforms block far more: 54.8% of the 124 counted apart block a training bot by name and 32.3% block a search or answer bot. Ranking position makes little difference, 12.8% of top-3 sites against 11.8% of sites at positions 4 to 10, while page type matters more: comparison-page sites block training bots at 16.4%, against 7.9% for home-page sites.

What the numbers do not prove

Seonix is careful about the limits of its own data. The sample is not the whole web: it comes from commercial searches chosen for 23 sites that use Seonix, plus DuckDuckGo, so sites that rank elsewhere may behave differently. A robots.txt file is a request, not a fact: bots may ignore it, and a site can also block bots at the firewall, which this study cannot see. Sites with no file are kept apart from the shares. A template also says little about intent: the study can see that a file carries the Cloudflare block, but not who turned it on or why. And this is a single crawl on a single day, so it shows no trend. The study also found 3,604 of 14,875 sites that answered clearly, or 24.2%, serve a valid llms.txt file, though it did not measure whether any AI engine uses it.

Why it matters for SEOs and link builders

For site owners, the practical lesson is to check your own file for the difference between the two kinds of bot. A site can refuse training and still stay visible in AI answers, because answer crawlers are different bots with different names. Blocking a search or answer bot is what can keep you out of those answers, and only 2.3% of sites do it. If your robots.txt carries the Cloudflare managed block, know it blocks eight training crawlers at once, and check whether that is really what you want. For link builders, the numbers are reassuring in one direction and cautionary in another: the pages you earn links on are overwhelmingly still open to the bots that bring them into AI answers, so a link's citation value is rarely killed by a robots.txt rule. But the 12.2% that block training bots are a reminder that content can be walled off from the datasets that train future models. Read your own file before you assume which side you are on.

Sources

Sources: Seonix, November 3, 2026.

Related reading on this site: Google: Technical SEO Remains Key for AI SEO, Cloudflare CEO: AI Agent Optimization Is the New SEO, Brave Sells Its Search Index to AI — Your Site Is Probably In It, Google Explains How Redirects Affect Ranking Signals, SEO Split Testing in 2027: How to Prove a Change Actually Worked, and AI Overviews Appear on 70% of Everyday UK Searches, Sayu Study Finds.

More from Linkbaits

Turn this playbook into links

Linkbaits helps you create content that earns citations and backlinks. Start with the toolkit and build your first linkable asset this week.

Start free trial