Search Engine Journal published a robots.txt guide on November 18, 2026 that separates AI training crawlers from search crawlers and user-requested visits. The distinction matters because the three kinds of visits need different rules, and a site that blocks one by accident can lose search traffic, AI citations, or both. The guide arrives as crawler control becomes a basic technical SEO skill.
The guide covers best practices for robots.txt as it applies to search and AI crawlers. The core idea is that not all crawlers are the same: search crawlers index your pages for results, AI training crawlers ingest your content to train models, and user-requested visits happen when someone asks an AI assistant to fetch a page for them. Each one deserves its own rule.
The distinction between training and user-requested visits is the part most site owners miss. Blocking an AI training crawler does not necessarily block an AI assistant from fetching your page when a user asks it to, and allowing a training crawler does not mean you have consented to every use of your content. The guide walks through how to set each control separately.
Google gives site owners separate switches for AI. The Search generative AI control in Search Console removes a site from AI Overviews and AI Mode, and it has been available to all websites worldwide since August 31, 2026. Google-Extended in robots.txt limits the use of content to train future Gemini models. On October 30, Google said it plans to support a shared AI training opt-out in robots.txt in 2027.
The shared opt-out is the one to watch. If the major AI companies adopt a single standard, site owners will be able to express a training preference once instead of maintaining a separate rule for every crawler. Until then, robots.txt is a patchwork, and a guide that maps it is genuinely useful.
For SEOs, robots.txt is now a strategic document, not a technical afterthought. The choices in it determine whether a site appears in AI Overviews, whether its content trains models, and whether AI assistants can fetch its pages. A wrong rule can cost a site its AI visibility, and a missing rule can cost it control over its content.
For link builders, the crawler rules affect the pages they pitch. A page that blocks AI crawlers may still rank in classic search, but it will not be cited in AI answers, which makes it less attractive as a link target. Knowing a prospect's robots.txt setup before pitching saves wasted outreach.
Watch for Google's shared AI training opt-out in 2027, which would simplify the whole picture. If it ships, the robots.txt guide will need a rewrite, and the sites that update first will have the cleanest control.
Watch also for a study of who actually blocks AI crawlers. A robots.txt study of 1,000 sites was published the same week, and its findings will show how many sites have acted on the guidance and how many have not.
The most common robots.txt mistake in the AI era is blocking a crawler by accident. A site owner who copies a blocklist from a blog post can end up blocking a search crawler along with a training crawler, and the result is a loss of search visibility that nobody notices until traffic drops. The guide's value is in preventing that mistake, by showing which rule does what and which crawlers each rule affects.
The second mistake is the opposite: doing nothing and assuming the default is safe. The default is to allow everything, and that means your content can train models, appear in AI answers, and be fetched by assistants. That may be what you want, but it should be a decision, not an oversight. A robots.txt file that reflects a deliberate choice is worth more than one that reflects neglect.
Sources: Search Engine Journal, November 18, 2026.
Related reading on this site: AI Overviews Showed Up in Only 20% of Local Searches, and Mostly Copied the Map Pack, ChatGPT Wallet Lets AI Agents Pay Without Seeing Your Card Number, November 2026 Core Update Still Rolling Out as Volatility Peaks on November 14, Domain Authority Barely Correlates With AI Citations, Search Atlas Finds, Social Networks Are a Minor AI Search Source — Except LinkedIn and YouTube, and Semrush Rebuilds Traffic and Market Data With a New Model, and Old Numbers Will Change.
Linkbaits helps you create content that earns citations and backlinks. Start with the toolkit and build your first linkable asset this week.
Start free trial