Technical SEO

Who Blocks AI Crawlers? A Robots.txt Study of 1,000 Sites Has Answers

A robots.txt study of 1,000 sites asks a simple question: who blocks AI crawlers? The study, published by DevToolry and flagged in Search Engine Roundtable's November 18, 2026 recap, reads the robots.txt files of a thousand sites and records which AI crawlers they block, which they allow, and which they ignore entirely. The pattern that emerges says a lot about who is paying attention to AI and who is not.

What the study measured

DevToolry's study examines the robots.txt files of 1,000 sites and records their treatment of AI crawlers. The study distinguishes between different crawlers, because blocking one AI company's crawler does not block another's, and between training crawlers and search crawlers, because they serve different purposes.

The method is straightforward: read the file, record the rules, count the results. There is no survey and no self-reporting, which makes the data more reliable than asking site owners what they do. The robots.txt file is the ground truth of a site's crawler policy.

Why the results matter

The study matters because crawler blocking is becoming a strategic decision, not a technical default. A site that blocks AI training crawlers is choosing not to let its content train models. A site that blocks AI search crawlers is choosing not to appear in AI answers. A site that does nothing is choosing by default, and the default is to allow everything.

The pattern across a thousand sites shows how far the industry has come. A year ago, most sites had no AI crawler rules at all. Now the question is which rules they have, and the answer varies widely by industry, site size, and business model.

Why it matters for SEOs and link builders

For SEOs, the study is a benchmark. A site owner can compare their own robots.txt against a thousand others and see whether they are ahead, behind, or typical. The benchmark is useful because crawler policy is one of those decisions that feels hard to judge in isolation.

For link builders, the study affects outreach. A site that blocks AI crawlers may still be a good link target for classic search, but it will not be cited in AI answers, which reduces its value as a source. Knowing a prospect's crawler policy before pitching saves time and sets expectations.

What to watch next

Watch for the full breakdown by crawler and by industry. The headline is the study's existence; the value is in the detail of which crawlers are blocked most and which industries block most aggressively.

Watch also for the next study. Crawler policies are changing fast, and a study from today will be outdated in six months. The trend matters more than the snapshot: are more sites blocking, or fewer, and which way is the industry moving?

The decision every site owner faces

Every site owner now faces a decision that did not exist two years ago: what should AI be allowed to do with my content? The options are not just allow or block. A site can allow search crawlers and block training crawlers, allow AI answers but not model training, or allow everything and accept the trade. Each choice has a cost, and the cost depends on the site's business model.

A news publisher that depends on subscriptions has a different answer from a software company that wants its documentation cited everywhere. The study of a thousand sites shows that there is no consensus yet, and that is the point: the industry is still deciding. The site owners who decide deliberately, and revisit the decision as the technology changes, will be better off than those who never decide at all.

Sources

Sources: DevToolry and Search Engine Roundtable, November 18, 2026.

Related reading on this site: AI Overviews Showed Up in Only 20% of Local Searches, and Mostly Copied the Map Pack, ChatGPT Wallet Lets AI Agents Pay Without Seeing Your Card Number, November 2026 Core Update Still Rolling Out as Volatility Peaks on November 14, Domain Authority Barely Correlates With AI Citations, Search Atlas Finds, Social Networks Are a Minor AI Search Source — Except LinkedIn and YouTube, and Semrush Rebuilds Traffic and Market Data With a New Model, and Old Numbers Will Change.

More from Linkbaits

Turn this playbook into links

Linkbaits helps you create content that earns citations and backlinks. Start with the toolkit and build your first linkable asset this week.

Start free trial