Get started
This guide instructs you on how to:
- View AI crawlers that are interacting with pages in your domain (a Cloudflare zone).
- Use AI Audit to block individual crawlers from accessing your content.
- Sign up for a Cloudflare account ↗.
- Connect your domain to Cloudflare.
- Make sure your domain is proxying traffic through Cloudflare.
To only block specific AI crawlers:
- Go to AI Audit.
- From the AI Crawlers tab, go to the list of AI crawlers.
- Review the AI crawlers that are accessing your domain.
- From the Block column, select the option Block.
To only block specific AI crawlers:
- Go to AI Audit.
- From the AI Crawlers tab, go to the list of AI crawlers.
- Review key information such as:
- Bot operator
- Number of requests sent by the AI crawler
- Whether the AI crawler is allowed according to your
robots.txt
, or the number of times the AI crawler has violated yourrobots.txt
.
- From the Block column, toggle the switch for AI crawlers you wish to block.
For more information, refer to Manage AI crawlers.
You can also create more complex rules when taking action on AI crawlers, using Cloudflare WAF. For more information on creating more specific rules, refer to Create a custom rule in the dashboard.
Review the AI crawlers detected on your site.
- Go to AI Audit.
- From the Metrics tab, view key metrics on AI crawlers.
Note that on free plans, the Metrics tab only displays metrics for the past 24 hours.
Refer to the following related resources:
- Cloudflare blog: Start auditing and controlling the AI models accessing your content ↗
- Block AI crawlers that do not adhere to recommended guidelines using Cloudflare AI Labyrinth.
- Direct AI crawlers with managed robots.txt.
Was this helpful?
- Resources
- API
- New to Cloudflare?
- Products
- Sponsorships
- Open Source
- Support
- Help Center
- System Status
- Compliance
- GDPR
- Company
- cloudflare.com
- Our team
- Careers
- 2025 Cloudflare, Inc.
- Privacy Policy
- Terms of Use
- Report Security Issues
- Trademark