Who Decides Who I Publish For?
Recently, a client decided to move her websites away from my hosting service. The reason was simple: I block a number of AI crawlers by default.
She believes — and I don’t think it’s an unreasonable belief — that websites should be accessible to as many language models as possible. The more AI systems can consume that content, the greater the chances that it will appear in AI-generated answers, gain visibility, and — hopefully — attract more visitors. It’s a perfectly understandable line of reasoning.
That experience, however, made me reflect on something that seems to be increasingly common and increasingly overlooked: who decided that I have an obligation to publish my content for anyone who wants to consume it?
I’m not talking about hiding content behind paywalls (not that there’s anything wrong with that), nor about turning the web into a collection of walled gardens. I’m certainly not arguing against artificial intelligence. I use AI almost every day, whether for writing, programming, or research. The point is that there is an enormous difference between making content available for humans to read and making it available for automated consumption at scale.
For a long time, that distinction barely mattered because there was an unwritten understanding: search engines indexed web pages and, in doing so, consumed server resources. In return, they sent visitors back to those sites, who might read more articles, buy a course, click on an ad, or otherwise generate value. There was a relationship of reciprocity. There was a cost, certainly, but there was also a clear benefit.
Today, that relationship is different. Language models are not designed to send people to your website; they are designed to answer questions so that people don’t need to visit it in the first place. I’m not even making a value judgment about whether that’s good or bad. I’m simply pointing out that the nature of the relationship has changed. And when the nature of a relationship changes, perhaps it’s worth rethinking assumptions that once seemed self-evident.
There is another aspect of this discussion that almost always gets overlooked: infrastructure costs money.
That sounds obvious, but it’s worth remembering that every request made to a website consumes CPU time, memory, bandwidth, disk I/O, and electricity. When a crawler traverses thousands of pages, someone is paying for those resources. And — spoiler alert — it isn’t the company operating the crawler: it’s the person paying for the server.
Curiously, almost every discussion about “opening websites to AI” seems to assume that these costs simply don’t exist, as if hosting websites were somehow a charitable activity, or as if CPU time and bandwidth were infinite resources. They aren’t, and for anyone who manages websites or works with infrastructure, those costs can be significant.
Does that mean nobody should allow AI crawlers? Of course not. That would be far too simplistic.
If you believe it makes sense to allow every language model to access your content, that’s a perfectly legitimate decision. I only think it should be a conscious one. After all, someone has to pay for the infrastructure required to support that decision. If you’re willing to pay that bill, that’s perfectly fine. Just don’t suggest that everyone else has a moral obligation to make the same choice.
There’s another interesting shift in perspective: for decades, we’ve considered it perfectly normal to decide who gets access to our systems. We have websites that require authentication, subscription-only services, geographically restricted content, authenticated APIs, firewalls, rate limits, and blocks based on IP addresses, countries, or any number of other criteria. All of these are perfectly ordinary aspects of running services on the Internet. I’ve never heard anyone argue that requiring authentication for an API is somehow “against the Internet.”
Likewise, I’ve never found it controversial for a website to block scrapers, spambots, or any other automated system that consumes resources abusively. Yet as soon as that automated system happens to be an AI crawler, blocking it suddenly becomes an act of technological sabotage. I find that change in perception rather curious.
It also wouldn’t be fair to lump all AI crawlers together. Some projects identify themselves properly, respect robots.txt, implement reasonable rate limits, and clearly document how they expect to be treated. Unfortunately, they do not represent the entire industry, and many companies preach one thing while doing another.
Today it is increasingly common — arguably the norm — to encounter crawlers that ignore robots.txt, distribute requests across thousands of IP addresses, lease residential proxy networks to blend in with ordinary visitors, constantly rotate their identities, or simply masquerade as legitimate browsers to evade basic filtering. Some companies go to great lengths to make their crawlers indistinguishable from human visitors precisely because they want to make blocking them more difficult.
It’s worth remembering that robots.txt was never intended to be a security mechanism. It was a mechanism for cooperation. Technically speaking, it never prevented anyone from accessing a website; it merely said, “I’d rather you didn’t do this.” For decades that worked because the major search engines respected the convention.
When a significant portion of the new generation of crawlers no longer considers those conventions worth respecting, it shouldn’t surprise anyone that administrators have begun replacing voluntary cooperation with technical enforcement.
That is precisely why projects such as Anubis and Iocaine came into existence. For some time I described them as tools for blocking AI. I no longer think that’s a fair description. They don’t exist to block AI; they exist to give administrators back a choice that always belonged to them, but somehow became politically inconvenient once the interested parties were AI companies.
Perhaps the most common argument is that blocking these crawlers is a strategic mistake because the future belongs to AI. That may well be true. It’s entirely possible that, a few years from now, language models will become the primary way people discover content on the web. If so, that’s perfectly fine.
The difference is that, unlike a search engine, a language model is not designed to direct users toward the original source. It is designed to answer the question directly. That’s simply how the technology works, and there’s nothing inherently wrong with it. The problem is that this fundamentally changes the economic relationship between those who produce content and those who consume it.
There is also an irony that’s hard to ignore. When an AI company considers the content of a website valuable enough to help generate billions of answers, it seems perfectly natural that it should be allowed to access it. Yet when the administrator of that same website decides not to participate in that arrangement, they are suddenly portrayed as someone who “doesn’t understand the future.”
Perhaps we’ll eventually see licensing models, compensation mechanisms, or genuinely respected consent protocols. I sincerely — and perhaps somewhat idealistically — hope so. Not because I expect to live in a perfect Internet, but because it seems difficult to sustain indefinitely a model in which one side produces the content, pays for the infrastructure, and bears all the costs, while the other unilaterally decides it has the right to consume both.
Perhaps I’m completely wrong, and ten years from now blocking AI crawlers will seem just as strange as blocking Googlebot did in the early 2000s. That’s certainly possible. Until costs and benefits become even remotely balanced again, however, I find it difficult to accept the idea that this relationship is somehow inevitable.
And I continue to believe that anyone who publishes a website should retain one very simple right: the right to decide who they are publishing it for.