The "Cyber Bodhisattva" strikes back: The era of "free lunch" for AI giants comes to an end
TechFlow Selected TechFlow Selected
The "Cyber Bodhisattva" strikes back: The era of "free lunch" for AI giants comes to an end
From "Firewall" to "Mosquito Net," Cloudflare might rewrite the history of paid web content.
Navigating Web3 tides with focused insights
Musk and Trump—the White House's dynamic duo—have recently escalated their verbal sparring into version 2.0. Similarly, the relationship between major publishing groups and AI giants is one of love-hate: some publishers are partnering with AI companies, while others are determined to sue them into bankruptcy.
Data shows that since the emergence of AI search and ChatGPT, global website traffic has declined. Meanwhile, AI giants’ “AI crawlers” ignore robots.txt protocols, scraping data from websites tens of thousands of times over.
Now, finally, an infrastructure company has stepped forward, taking content creators by the hand and declaring: "We can say no to AI giants!"
Cloudflare, the internet infrastructure giant that handles about 20% of global web traffic and is hailed by netizens as the "cyber bodhisattva," launched an experimental product and marketplace in July 2025: "Pay Per Crawl"—setting new rules for AI crawlers:
Either get permission, or pay up.
In short, this feature gives website content creators a toggle option: allow free access to AI crawlers, charge per crawl, or block access entirely.
As Cloudflare’s co-founder put it, “Content is the fuel powering AI engines, so it’s only fair that content creators are directly compensated.”
For AI companies, the era of freely feasting on web content to train models is over. But it’s not all downside—by paying transparent, fixed prices, they can avoid copyright disputes.
Can Cloudflare’s anti-crawler move curb the rampant abuse of AI bots? More importantly, can this company leverage its unique position to create an entirely new model for content distribution and monetization in the AI era?
01
The AI Giants’ Free Lunch
For decades, most websites have been publicly crawlable by default. Search engines like Google and Bing drove traffic to sites, which then monetized via ads or subscriptions—this was the unspoken contract of the search era.
But in the AI era, traditional search traffic has plummeted, making this equation increasingly unsustainable.
AI companies treat web content as training fuel, yet offer little or no compensation to most creators. When users ask questions directly to AI chatbots, answers are often synthesized summaries rather than lists of blue links—meaning websites receive no referral traffic.
Even search giants like Google are shifting: instead of providing link lists, they now offer “AI Overviews.” According to their reports, 75% of queries are resolved without users clicking any links.
Cloudflare’s latest data from July 2025 shows: Google’s crawler generates one click back to a site every 6–7 fetches, OpenAI manages one return click per 1,500 crawls, and Anthropic’s ratio is even more extreme—at 73,300 crawls for just one redirect.
Click-through rates per crawl by major AI companies｜Image source: Cloudflare
This means the old “content for traffic” model is broken. Compared to traditional search engines, AI giants consume vast amounts of website content but provide no traffic in return. This imbalance is making it harder for content producers to survive.
“With OpenAI, acquiring traffic is 750 times harder than in the Google era; with Anthropic, it’s 30,000 times harder. The reason is simple: we’re consuming less original content and more of its derivatives,” wrote Cloudflare CEO Matthew Prince in a blog post. “This is not a fair exchange.”
There are costs for AI companies too. In recent years, AI giants have faced widespread accusations of “stealing content” to train large models, triggering a wave of global copyright lawsuits—especially from media outlets like The New York Times against OpenAI.
Active AI crawler bots from major tech firms｜Image source: Cloudflare
Hence, Cloudflare introduced “Pay Per Crawl”—a “pay-per-fetch” marketplace designed to solve this very issue.
The company built a permissions and payment system allowing websites to choose whether to allow, block, or charge AI crawlers. If an AI bot wants to scrape content, it must register, verify its identity, and complete a payment for each visit.
If successful, this model could shift web content monetization from ad-based to licensing-based revenue, opening new income streams. Whether major media or niche blogs, all content creators could gain pricing power and be paid for AI usage.
To emphasize its significance, Cloudflare’s CEO dubbed the launch day of “Pay Per Crawl”:
“Content Independence Day.”
02
How Is the AI “Toll Fee” Collected?
Ideas are great, but how does it work technically?
Cloudflare started by offering CDN, DDoS protection, DNS, and zero-trust security services. With nodes deployed in over 300 cities worldwide, handling about 20% of web traffic, it’s well-positioned to act as an intermediary.
“Pay Per Crawl” operates at the middleware layer of Cloudflare’s global CDN network—enabling it to identify and process AI crawler requests before they reach the origin server. Website admins can set one of three modes in the Cloudflare dashboard: allow, charge, or block.
Admins can configure allow, charge, or block settings｜Image source: Cloudflare
All new websites joining Cloudflare are defaulted to block AI crawlers unless the admin explicitly allows access. Only AI companies partnered with Cloudflare can participate in the payment system; others will be blocked.
If an AI crawler attempts to access a paid URL without payment, Cloudflare returns an HTTP 402 Payment Required status code—a long-ignored code originally reserved for online payments. The crawler can include payment information in the request header to accept the price. Once verified, access is granted with a 200 OK response, and automatic settlement occurs.
Cloudflare acts as the “cashier,” aggregating bills and distributing earnings.
Cloudflare returns HTTP 402 Payment Required｜Image source: Cloudflare
Crawlers can include payment info in the request｜Image source: Cloudflare
HTTP 200 OK confirms payment and access｜Image source: Cloudflare
Crucially, this system cannot be bypassed by simple User-Agent spoofing. Cloudflare requires AI companies to register cryptographic keys, using digital signatures to authenticate identity—preventing fake crawlers from evading fees.
In the past, robots.txt—a plain text file in a website’s root directory—was used to indicate which pages search engine crawlers could access. But it was merely a polite suggestion, often ignored by AI scrapers. Cloudflare changes this by turning the soft “robots.txt” guideline into a hard gatekeeper.
Notably, according to Cloudflare, only about 37% of the top 10,000 domains currently have a robots.txt file.
Setting barriers for AI crawlers｜Image source: Cloudflare
To join Cloudflare’s crawling marketplace, both crawlers and website owners must have Cloudflare accounts. Currently, “Pay Per Crawl” is in private beta, with only select major publishers participating—including BuzzFeed, The Atlantic, and Fortune. Cloudflare continues to solicit interest from other content creators and crawlers.
“We expect the pay-per-fetch model to see significant growth,” Cloudflare stated officially.
Though still early, the company has ambitious future plans: publishers could set different prices for different content types, dynamically price based on an AI app’s user base, or introduce granular pricing for training, inference, and search use cases.
They also believe the true potential of pay-per-crawl may emerge in the world of AI agents.
“What if agent paywalls could operate fully programmatically? Imagine asking your deep research assistant to compile the latest cancer studies, legal briefs, or find the best restaurant—and giving the agent a budget to access the most useful, relevant content.”
“The first solution built on the HTTP 402 response code will enable a future where intelligent agents can programmatically negotiate access to digital resources,” Cloudflare said.
03
At the Crossroads of the Internet
Economically, this could mark the beginning of renegotiating revenue shares between AI and content creators.
Currently, only top-tier media outlets (like The New York Times, which sued OpenAI before settling) can negotiate licensing deals. The vast majority of small websites, forums, and individual authors are silently scraped—with neither ability nor awareness to resist. Cloudflare’s solution democratizes this bargaining power across the web.
According to the Cloudflare team, after hundreds of conversations with news organizations, publishers, and large social platforms, there’s a consensus: “We want to allow AI crawlers to access our content—but we want to be paid.”
To supporters, the “Pay Per Crawl” model is conceptually fair: creators earn revenue, AI companies reduce legal risk, and long-term, it promotes a more compliant content licensing ecosystem.
Image source: Cloudflare
Of course, AI companies may not be thrilled. Web data is no longer free; accessing fresh content now comes at a cost—adding a new expense beyond compute power.
Yet, this might also curb indiscriminate scraping, forcing AI developers to be more selective—buying high-value content strategically instead of dumping everything into their models.
Matthew Prince argues, “AI engines are like Swiss cheese—what truly fills the holes are novel, original contents, far more valuable than the repetitive, low-quality content dominating much of today’s web.”
In his view, traffic has never accurately reflected content value. “If we start evaluating content not by how much traffic it generates, but by how much it advances knowledge—measured by how many gaps it fills in AI engines—we could not only accelerate AI progress but potentially usher in a golden age of high-value content creation.”
Still, digital rights advocates raise concerns: Can small AI startups, researchers, and open-source communities afford these data costs? Will benevolent crawlers for academic research or public archiving be crippled, limited to low-value, restricted sources?
In a world where ad revenue is falling and traffic acquisition is costly, how many websites will willingly let AI bots feed for free? Could this become the beginning of internet closure, eroding its spirit of openness and sharing?
If the entire web defaults to blocking and charging, could this inadvertently strengthen big tech monopolies? After all, big players can afford to pay.
The “Pay Per Crawl” model tries to fix AI’s one-sided consumption of content, but may unintentionally raise the barrier to AI innovation—reviving the old tension between copyright protection and open knowledge.
Still, Cloudflare only expands choice. Website owners remain free to grant free access to non-profits or public-interest projects. The power stays with creators. Either way, they deserve compensation.
As Cloudflare’s CEO puts it, the goal of this transformation is “building a better internet.” “We don’t have all the answers yet, but we’re working with leading economists and computer scientists to find them.”
So far, other CDN and security providers (like Akamai, Fastly, Amazon CloudFront) have not announced similar features.
Blocking AI crawler bots at the door｜Image source: Cloudflare
While “Pay Per Crawl” may appear to be just a new CDN feature, in a broader sense:
It could signal a pivotal moment for the internet.
In the search era, content value was converted into ad revenue through user visits. But in the AI era, users may never visit websites—the answers are all generated within chatbots. Should large AI models continue mining web content for free, or should data access return to a principle of reciprocity, ensuring creators are fairly compensated? And how much should that compensation be?
This early experiment may be laying the groundwork for a new data economy in the AI age. Regardless of success or failure, its stance is clear: AI cannot indefinitely exploit creators’ patience, using “openness” as justification to turn human labor into free fuel.
“The web is transforming, and its business model must evolve too. In this process, we have the chance to learn from the past 30 years and make the internet better for the future.”
As for whether things will actually improve—as Cloudflare itself admits:
“This is just the beginning.”
Join TechFlow official community to stay tuned
Telegram:https://t.me/TechFlowDaily
X (Twitter):https://x.com/TechFlowPost
X (Twitter) EN:https://x.com/BlockFlow_News
极客公园
