On 29 October 2026 Vercel added an option to its firewall that sounds small but changes a decision many site owners have been making with one blunt switch. Its AI Bots managed ruleset can now treat two kinds of AI traffic differently. In Vercel's words:
- "AI crawlers: bots that collect content for indexing and model training"
- "AI assistants: bots that fetch pages in response to user actions or search queries"
And the example Vercel gives: "you can block AI crawlers while allowing AI assistants to retrieve your content to answer users' questions." Vercel says the ruleset is "available on all plans" and that it "updates the rules as new bots are added."
You don't need to use Vercel for this to matter. The split it draws is a useful way to think about AI traffic on any website.
Why one switch was the wrong shape
Until now, the choice was mostly all or nothing. If you didn't want your content collected for model training, blocking AI bots also kept your pages from being fetched when someone asked an assistant a question your site could answer. If you wanted to be found that way, you let everything in.
Those are different trade-offs, and a business can reasonably want one without the other.
Three questions to decide the policy
- Do you want your content in training data? For a public product page, maybe you don't mind. For original research, paid content or material that's your competitive edge, you may well mind.
- Do you want to be fetched when someone asks an assistant? If customers ask an AI tool about what you sell, being retrievable at that moment may matter more to you than the training question.
- Is the answer the same for every part of the site? Often it isn't. A blog you want widely read and a members' area you don't are different decisions.
Those are business decisions. The firewall setting is just how you carry them out.
Practical points before you flip anything
- Look before you block. Vercel's own example blocks crawlers but sets assistants to log rather than block. Logging first shows you how much of each kind of traffic you actually get, before you decide.
- The categories are the vendor's classification. Which bot counts as a crawler and which as an assistant is decided by whoever maintains the rules, and the changelog doesn't list them. Check the documentation and your own logs rather than assuming a particular bot lands where you expect.
- Write the policy down. A short note saying what you allow, what you block and why makes it easy to revisit as the bots change, and keeps the decision from living only in a dashboard setting.
- Revisit it. New bots appear, and your goals change. A choice made today is worth checking again in a few months.
The useful change here isn't the button. It's that "keep my content out of training" and "let assistants find me" have become two separate decisions, and they deserve two separate answers.
Source: Vercel changelog, "Control AI crawlers and assistants separately in Vercel Firewall" (29 October 2026, read 31 October 2026).