The Opt-Out Illusion: Why Retreating from AI Indexing Cannot Save Copyright
The seductive logic of the “no AI robot” option is that it mirrors the familiar robots.txt control that once seemed to govern search indexing. If a creator does not want their work ingested by AI, the reasoning goes, they simply flip the switch, publish the directive, and the law will back them up. This is a dangerous fiction. The AI opt-out is not a lock on the door; it is a note taped to a window, and the legal and technical architecture surrounding it ensures that the note can be ignored, stripped away, or rendered meaningless without consequence.
A sign, not a shield. The most direct refutation of the opt-out’s legal efficacy came in December 2025, when the Southern District of New York ruled in the Ziff Davis v. OpenAI litigation that a robots.txt file is not a “technological measure that effectively controls access” to copyrighted works under the Digital Millennium Copyright Act. The court’s analogy was devastatingly simple: a robots.txt directive controls access to a website “any more than a sign requesting that visitors ‘keep off the grass’ effectively controls access to a lawn”. If disregarding the file does not constitute circumvention, then the opt-out carries no statutory enforcement mechanism in the United States. It is a request, not a right.
The inversion of consent. The European framework is no more reassuring. Article 4 of the DSM Directive permits commercial text and data mining by default unless the rightsholder has expressly reserved their rights in a machine-readable format. This structure places the burden of objection on the creator, transforming silence into permission. Yet the mechanism for expressing that objection remains deeply underdeveloped. Courts across Europe have struggled to articulate what “machine-readable” actually requires, with decisions pointing in markedly different directions. The opt-out is not a recognition of consent; it is the non-exercise of a statutory veto over a use that the law already permits. As the UK’s own experience demonstrates, the political consensus behind this model has collapsed. In March 2026, the UK government formally abandoned its previously “preferred option” of a text and data mining exception with a rightsholder opt-out, acknowledging that the regime was unworkable and endorsing no alternative. Of the 11,520 consultation responses, 81 per cent favoured mandatory licensing; only 3 per cent supported the opt-out model.
The infrastructure strips the signal. Even if a creator publishes a valid opt-out, they cannot control what happens to it downstream. In September 2026, Cloudflare—one of the largest content delivery networks—migrated its systems and, in the process, stripped AI opt-out directives from the robots.txt files of 118 sites that had carried them. Cloudflare itself concedes the deeper problem: “A robots.txt directive alone cannot solve this problem. Anyone can publish one, but it cannot identify who is crawling, determine why they are crawling, or stop a crawler that ignores it”. The infrastructure layer that mediates between creators and crawlers is not neutral. It can alter, remove, or override the very signals on which the opt-out model depends.
The fragmentation trap. The opt-out landscape is further fractured by the proliferation of crawler identities. Blocking GPTBot does not block OAI-SearchBot; opting out through Google-Extended does not remove a site from Google’s AI Overviews, which ride the ordinary Googlebot. A study of four million citations across 3,600 prompts found that 92.3 per cent of top news sites blocking Google-Extended still appeared in AI citations. Creators are asked to navigate a shifting maze of user-agent tokens, each governed by different operators with different commitments, while the consequences of blocking are themselves asymmetric: research has associated blocking AI crawlers with a loss of roughly 7 per cent of weekly traffic within six weeks, a cost borne by the creator while the citation persists.
The retreat that solves nothing. Under these conditions, the decision to withdraw from AI indexing entirely is not an overreaction; it is a rational act of self-preservation for those who can afford the visibility loss. But it is a strategy of retreat, not of rights management. It protects the individual who opts out while leaving the default extraction of the open web untouched. The real necessity is not the perfection of opt-out tags or the proliferation of crawler-specific directives. It is a shift toward a licensing-first framework in which the burden of permission sits with the entity that wishes to use the work, not with the creator who must constantly announce what they forbid. Until that shift occurs, the “no AI robot” option remains what the Ziff Davis court implied: a sign on an unlocked door, addressed to visitors who have already shown they may not be reading.
