A robots.txt study of 1,000 sites asks a simple question: who blocks AI crawlers? The study, published by DevToolry and flagged in Search Engine Roundtable's November 18, 2026 recap, reads the robots.txt files of a thousand sites and records which AI crawlers they block, which they allow, and which they ignore entirely. The pattern that emerges says a lot about who is paying attention to AI and who is not.
DevToolry's study examines the robots.txt files of 1,000 sites and records their treatment of AI crawlers. The study distinguishes between different crawlers, because blocking one AI company's crawler does not block another's, and between training crawlers and search crawlers, because they serve different purposes.
The method is straightforward: read the file, record the rules, count the results. There is no survey and no self-reporting, which makes the data more reliable than asking site owners what they do. The robots.txt file is the ground truth of a site's crawler policy.
The study matters because crawler blocking is becoming a strategic decision, not a technical default. A site that blocks AI training crawlers is choosing not to let its content train models. A site that blocks AI search crawlers is choosing not to appear in AI answers. A site that does nothing is choosing by default, and the default is to allow everything.
The pattern across a thousand sites shows how far the industry has come. A year ago, most sites had no AI crawler rules at all. Now the question is which rules they have, and the answer varies widely by industry, site size, and business model.
For SEOs, the study is a benchmark. A site owner can compare their own robots.txt against a thousand others and see whether they are ahead, behind, or typical. The benchmark is useful because crawler policy is one of those decisions that feels hard to judge in isolation.
For link builders, the study affects outreach. A site that blocks AI crawlers may still be a good link target for classic search, but it will not be cited in AI answers, which reduces its value as a source. Knowing a prospect's crawler policy before pitching saves time and sets expectations.
Watch for the full breakdown by crawler and by industry. The headline is the study's existence; the value is in the detail of which crawlers are blocked most and which industries block most aggressively.
Watch also for the next study. Crawler policies are changing fast, and a study from today will be outdated in six months. The trend matters more than the snapshot: are more sites blocking, or fewer, and which way is the industry moving?
Every site owner now faces a decision that did not exist two years ago: what should AI be allowed to do with my content? The options are not just allow or block. A site can allow search crawlers and block training crawlers, allow AI answers but not model training, or allow everything and accept the trade. Each choice has a cost, and the cost depends on the site's business model.
A news publisher that depends on subscriptions has a different answer from a software company that wants its documentation cited everywhere. The study of a thousand sites shows that there is no consensus yet, and that is the point: the industry is still deciding. The site owners who decide deliberately, and revisit the decision as the technology changes, will be better off than those who never decide at all.
Sources: DevToolry and Search Engine Roundtable, November 18, 2026.
Related reading on this site: AI Overviews Showed Up in Only 20% of Local Searches, and Mostly Copied the Map Pack, ChatGPT Wallet Lets AI Agents Pay Without Seeing Your Card Number, November 2026 Core Update Still Rolling Out as Volatility Peaks on November 14, Domain Authority Barely Correlates With AI Citations, Search Atlas Finds, Social Networks Are a Minor AI Search Source — Except LinkedIn and YouTube, and Semrush Rebuilds Traffic and Market Data With a New Model, and Old Numbers Will Change.
Linkbaits helps you create content that earns citations and backlinks. Start with the toolkit and build your first linkable asset this week.
Start free trial