Whether you'd like AI companies to train on your website or not, you might assume your robots.txt already says so, or that a line blocking every crawler does. A proposal for a robots.txt training signal says, in its own words, that neither is true. This piece is about the details of that proposal and its traps, not about which choice is right. That's yours to make.
The proposal is an Internet-Draft called "Machine-Readable Training Preference Signal", written by K. Kelley of Anthropic and dated 30 October 2026. Before anything else, its status: it is an individual submission, revision 00, not an adopted standard. Its header says "Intended status: Standards Track", which is an intention, and it expires on 3 May 2027 unless it's updated. The draft's own boilerplate says "It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."" So read everything below as a proposal, not a rule anyone has to follow.
The mechanism
The draft doesn't add new robots.txt commands. It adds a special name you put where a crawler's name would normally go, and then uses the familiar Allow and Disallow lines. Its example:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/According to the draft, this says you'd prefer your content isn't used for AI training, except for anything under /press-kit/, where you're in favour of it. It works both ways: under the same name, Disallow expresses a preference against and Allow expresses a preference in favour, so a site that welcomes training can say that just as clearly. The draft takes its definition of AI training from the IETF's AI Preferences working group: "Using an asset to modify the learned parameters of a generative AI model."
Four details that are easy to get wrong
1. Blocking all crawlers doesn't count
Some sites already have a blanket rule, User-agent: * with Disallow: /. Under this proposal, that expresses no training preference at all. The draft is explicit: "However, wildcard groups (i.e., User-Agent: *) MUST NOT be interpreted as matching the policy token."
2. Saying nothing isn't saying no
"If no group contains the __usage_ai-training policy token (i.e., there is no User-Agent: __usage_ai-training line in the file), then no preference has been expressed." No line means no preference either way.
3. It doesn't block anything
This is a preference, not a barrier, as the draft states directly: "This policy token is not a directive for crawling behavior." The section on how it relates to crawler rules is titled "Independence from Crawler Rules", and puts it plainly: "Content-usage preferences and traffic management signals can both be expressed in robots.txt, but they are independent and orthogonal."
That matters if you've recently set up bot controls. We've written about Vercel's firewall controls for AI crawlers and assistants and Cloudflare's per-agent policies. Those control traffic. This signal is about what happens to content after it's been fetched. Setting one doesn't set the other.
4. Other rules can take priority
The draft allows two things to override this signal. Vendors can treat their own, vendor-specific robots.txt names as more specific than the general one, and the draft says "Private agreements MAY override preferences expressed via this mechanism." If you've signed a content deal, its terms may matter more than this line.
What it doesn't tell you
The draft describes how a preference would be written and read. It doesn't say which AI companies will honour it, and we haven't seen a primary statement from any of them committing to it. You may see reports saying particular companies plan to adopt it. Treat those as unconfirmed until the company says so itself.
The draft lists contributors from a range of organisations, including large technology companies, publishers and civil-society groups, but it also says "Affiliations are listed for identification only, since IETF participants contribute as individuals." A name in that list isn't a company's endorsement.
One more detail worth knowing: the draft says the preferences that apply to a piece of content are the ones in the robots.txt in effect when the content was retrieved. A line you add today speaks to future retrievals, not to anything already collected.
What to do
- Don't assume your current robots.txt says anything about training. Under this proposal, a wildcard block doesn't, and no line means no preference.
- Decide what you actually want. Training welcome everywhere? Nowhere? Welcome on some pages, like a press kit, but not others? The proposal lets you say any of these, per path.
- Keep traffic control and training preference separate in your head. Firewall and bot rules decide who can fetch your pages. This signal states a preference about use. You may want both.
- Watch the status. This is revision 00 of an individual draft. The name, syntax or meaning could change before, or if, it becomes a standard.
Do you know what your own robots.txt says today, and would it say what you mean under this proposal?
Source: K. Kelley, "Machine-Readable Training Preference Signal", Internet-Draft draft-kelley-aipref-training-preference-signal-00 (individual submission, published 30 October 2026, work in progress; read 7 November 2026).