‘Pay per crawl’: Major publishers back Cloudflare’s plan to charge AI companies
The day after OpenAI issued an economic blueprint for Australia that called for tax funding to boost AI, internet backend company Cloudflare has declared war on AI bot scrapers, with a new default bot blocking regime backed by major global publishers.
Cloudflare provides networking infrastructure that reportedly covers around 20% of all internet traffic. From today, it says it will block AI crawlers by default. The system allows publishers to charge AI models for accessing its content.
I also agree with Nick Shelton from Broadsheet that the future of media likely isn’t on AI and Government hand-outs and there’s a strong need for media to reinvent itself with stronger commercial models (subscriptions, courses, first party data, owned audiences) than being wholly reliant on the pageview economy or impression volumes.
Curious to see how this plays out in the real world. On paper the model sounds elegant, yet I can’t ignore a few practical hurdles:
> Negotiation bottleneck – Mid-sized publishers like ours can’t even get a read-receipt from OpenAI, Perplexity or Anthropic, let alone strike a Cost-per-Crawl deal. Without a collective marketplace or regulatory nudge, the biggest bots may simply scrape elsewhere.
> Nascent revenue – TollBit, Dappier and the rest show lots of 402 responses but very little actual cash. History suggests crawlers just bounce when a paywall pops up rather than flashing their corporate Amex and paying the fee to crawl
> Discoverability trade-off – Blocking non-paying bots risks shrinking our presence in AI answer boxes, chopping traffic before the cheques start rolling in.
> Flat-rate pricing limits – A single price across an entire domain feels blunt. Granular or dynamic rates will be essential if this is to scale.
Hopeful it succeeds because creators do deserve fair value for their original journalistic content, but I’d love a clearer roadmap of how Cloudflare (as well as Tollbit, Dappier, Prorata.ai ScalePost, IAB Tech Lab’s new LLM Content Ingest API Iniaitive) will turn good intentions and HTTP 402s into real, material income for the broader publishing sector beyond the larger legacy players and Condé Nasts of the world. Right now it feels like many vendors are racing to lock up market share ahead of the legislation.