As of the wire November 1, 2026, capture, NPR robots.txt disallows Meta meta-externalagent site-wide, according to the Anything Engine Optimization crawler-access panel. The panel had no entry for that crawler in NPR's file a week earlier, and NPR already blocked every other tracked AI crawler, with its Claude-SearchBot rule logged on September 27.
NPR position is now the strictest in the panel: every tracked AI crawler is blocked, with Meta agent the latest addition. That completeness is itself the story. Where The Atlantic chose disallow-some rules for OpenAI crawlers, NPR chose a site-wide disallow for Meta agent on top of existing blocks, leaving no tracked AI crawler with access.
The practical effect depends on how Meta uses the crawler. Meta-externalagent is associated with Meta AI training and data collection, so a site-wide disallow is a training-access decision more than a search-visibility decision. NPR content can still appear in classic search results; what changes is Meta AI's ability to collect it for training and grounding.
The NPR and Atlantic changes landed in the same capture week, and they follow the shared training-token announcement from the major AI companies days earlier. Publishers appear to be acting in two lanes at once: tightening robots files now with existing tools, while preparing for the shared standard that will let them set preferences once across companies.
That two-lane behaviour is rational. Robots changes take effect as crawlers re-read the file, while the shared token still awaits IETF process and company implementation through 2027. A publisher that waits for the standard leaves months of collection unaddressed; a publisher that only edits robots files will need to redo the work when the standard lands.
For SEOs, the wave of blocks cuts both ways. Blocked publishers lose AI citations, which removes a competitor from the answer, but also shrinks the pool of sources assistants can cite, which can make answers thinner or more reliant on aggregators. Sites that stay open and authoritative gain citation share by default as neighbours close their doors.
The strategic question for each publisher is therefore not whether to block, but what to block, for which use, and for how long. A blanket block is simple and defensible, but a targeted block that preserves search grounding while stopping training collection may preserve more value. NPR chose the blanket route; The Atlantic chose the targeted route, and both are now live experiments.
Every publisher now faces the same two-part decision NPR has answered in the strictest way: should AI companies be allowed to train on your content, and should they be allowed to ground answers in it? The two decisions have different costs. Blocking training protects long-term content value but gives up nothing immediate, since training does not send traffic. Blocking grounding gives up citations and the brand presence that comes with them.
Most publishers will land somewhere between NPR blanket block and The Atlantic targeted restrictions, and the right position depends on the content. Investigative reporting and unique data gain little from being summarised and may warrant training blocks, while service journalism and explainers may gain from grounding citations. The mistake is making one site-wide decision for all content types, instead of deciding section by section, with the commercial value of each section in mind.
Smaller publishers watching NPR should note that a strict block is easier to maintain than a nuanced one, but harder to reverse gracefully. A site-wide disallow can be lifted in one edit, yet the citations and relationships lost during the block do not return instantly. Before copying the strictest position, weigh how much of your current audience arrives through AI answers, and whether that audience has another route to reach you.
Sources: Anything Engine Optimization wire, November 1, 2026.
Linkbaits helps you create content that earns citations and backlinks. Start with the free tools and build your first linkable asset this week.
Start free trial