Cloudflare has introduced a new tool that enables website owners to block artificial intelligence (AI) crawlers from accessing their content without authorization or compensation. The move aims to help publishers regain control over their work as AI firms increasingly scrape online material to train large language models. With the “pay per crawl” feature, content creators can either deny access or set a price for it, Cloudflare said on Tuesday.
The initiative is supported by leading publishers such as Condé Nast and the Associated Press, along with social media platforms like Reddit and Pinterest. These companies have grown concerned about declining referral traffic from search engines, which previously fueled ad revenue. The new tool, Cloudflare says, could offer a sustainable path forward as AI-generated search answers reduce the need for users to click through to original sources.
Cloudflare’s Chief Strategy Officer, Stephanie Cohen, described the shift as critical, citing dramatic changes in traffic patterns. According to the company, Google’s referral ratio has dropped from 6:1 to 18:1 in six months, while OpenAI’s ratio stands at a staggering 1,500:1—indicating far less user redirection. AI-generated summaries, like Google’s AI Overviews, may be contributing to this trend, further cutting publishers out of traffic and monetization.
Tensions between AI companies and content creators continue to rise. While some firms, including the New York Times, have filed copyright lawsuits against AI developers, others have opted for licensing agreements. Reddit, for instance, is suing Anthropic for scraping user comments while simultaneously striking a deal with Google. Cloudflare’s tool marks a significant step in restoring balance to the digital content ecosystem.