Crawl pages
graph LR A[Set up your<br>Cloudflare Account] --> B[Connect to<br>Stripe] B --> C[Verify your<br>AI crawler] C --> D[Crawl pages]:::highlight classDef highlight fill:#F6821F,color:white
Once your AI crawler complies with Web Bot Auth, you can begin to crawl webpages. For more information on how pay per crawl works, refer to What is Pay Per Crawl?
When an AI crawler makes a request to a page protected by pay per crawl, the server will respond with HTTP/2 402 Payment Required
. This response will also include the crawler-price
header which specifies the cost to access the content. For example, a response may look like the following:
HTTP/2 200date: Fri, 06 Jun 2025 08:42:38 GMTcrawler-charged: USD 0.01
To access this content, the AI crawler must provide headers for paid access.
Include Web Bot Auth headers by following the steps Sign your requests
Your AI crawler can specify the price it is willing to pay by providing one of two headers:
crawler-exact-price
: This is the exact price the AI crawler is configured to pay for access. This value should exactly match the price set in the response headercrawler-price
. Include this header in your second request if your AI crawler first received a HTTP status code 402, and you wish to access the content by paying thecrawler-price
.crawler-max-price
: This is the maximum price the AI crawler is configured to pay for access on any content. Use this option if you wish to access any pay per crawl content with a crawl price equal or lower than the maximum price. If a page'scrawler-price
is higher than yourcrawler-max-price
, your AI crawler will receive a HTTP 402 response.
When you specify a header to indicate payment, you may receive one of two responses:
The value of the crawler-charged
header indicates the exact amount that will be billed to your Cloudflare account for the request.
HTTP/2 200date: Fri, 06 Jun 2025 08:42:38 GMTcrawler-charged: USD 0.01
If the request is unsuccessful due to an incorrect price header, you will receive a response with the crawler-price
header. Your request header values must be adjusted to match the crawler-price
.
HTTP/2 402date: Fri, 06 Jun 2025 08:42:38 GMTcontent-type: text/plain; charset=utf-8crawler-price: USD 0.01
When you successfully access pay per crawl content (response with HTTP status code 200), the response will include crawler-charged
. For example:
crawler-charged: USD 0.01
Cloudflare strongly recommends tracking and saving these values to keep an accurate record of the bill your AI crawler has accrued.
You may wish to refer to the following resources.
Was this helpful?
- Resources
- API
- New to Cloudflare?
- Products
- Sponsorships
- Open Source
- Support
- Help Center
- System Status
- Compliance
- GDPR
- Company
- cloudflare.com
- Our team
- Careers
- 2025 Cloudflare, Inc.
- Privacy Policy
- Terms of Use
- Report Security Issues
- Trademark