Cloudflare has launched the Pay per Crawl service, creating a content marketplace for authors and AI companies.
Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content. https://t.co/PUrlSs636k
— Cloudflare (@Cloudflare) July 1, 2025
As part of the experiment, website owners can allow or deny AI scanners access to their resources and set a fixed fee for each page crawl.
Additionally, they can offer free page analysis or block it entirely.
The tool allows users to see the purpose for which crawlers are viewing the site:
- for AI training;
- to provide answers via an AI search engine;
- for other purposes.
Cloudflare provides services for the protection, acceleration, and stable operation of websites and internet applications. The cloud infrastructure provider accounts for 20% of internet traffic.
The new service represents a significant idea that could offer a potential business model in the age of artificial intelligence. The launch comes at a time when news publishers face existential questions about reaching readers, Google Search traffic is declining, and users are opting for chatbot searches.
Some companies, like The New York Times, have filed lawsuits against tech firms for training their AI models on news articles without permission or payment. Others have struck multi-year deals to license content.
Cloudflare aims to create a more sustainable system where publishers can set prices on their terms.
The company also announced that all new websites created with Cloudflare will now block AI crawlers by default. If owners wish to allow automated data collectors access to their resources, they must grant permission. This gives each new domain “default control,” the firm asserts.
To test Pay per Crawl, AI companies and publishers need to have Cloudflare accounts. Both parties can set rates in their accounts at which they agree to buy and sell content.
Currently, cryptocurrencies are not involved in the new solution, although some experts believe that this asset class is ideally suited for such interactions.
In June, federal judge William Alsup ruled that Anthropic had the right to train models on published books without the authors’ permission.