FAQs
No. Pay per crawl allows setting different configurations (Block, Charge, or Allow) for each crawler, but it only supports setting a single price for all crawlers set to "Charge".
Every time your AI crawler accesses content in a website protected with pay per crawl, it will incur a cost, as set by the site owner. You must implement mechanisms within your crawler to track its expenditure and to enforce any desired spending limits you wish to set.
No. Charging events are only issued on successful HTTP response codes. Errors are not billed, even if you have sent the crawler-exact-price
header or the crawler-max-price
header.
You should use the normal user agents associated with your AI crawler that you have onboarded to Cloudflare and identified using Web Bot Auth.
Was this helpful?
- Resources
- API
- New to Cloudflare?
- Products
- Sponsorships
- Open Source
- Support
- Help Center
- System Status
- Compliance
- GDPR
- Company
- cloudflare.com
- Our team
- Careers
- 2025 Cloudflare, Inc.
- Privacy Policy
- Terms of Use
- Report Security Issues
- Trademark