Skip to content
Cloudflare Docs

AI Audit with Cloudflare WAF

AI Audit works alongside other Cloudflare products, such as Cloudflare Web Application Firewall (WAF). WAF checks incoming web and API requests, and filters undesired traffic based on rules. WAF custom rules allow you to perform certain actions such as enforcing robots.txt.

Order of precedence

  • AI Audit uses WAF custom rules to block the selection of AI crawlers the site owner has decided to block.
  • AI Audit's pay per crawl feature takes place after WAF.
graph LR
A[Traffic] --> B[WAF custom rules<br>AI Audit: Crawler blocks]
B --> C[Cloudflare<br>Bot Solutions]
C --> D[AI Audit:<br>Pay Per Crawl]
classDef highlight fill:#F6821F,color:white

For this reason, if you plan on using AI Audit to manage AI crawlers, you may wish to modify your existing WAF custom rules such that it does not affect AI crawlers. This will allow you to manage AI crawlers only from AI Audit, thereby streamlining your workflow.

Examples of using WAF vs AI Audit

Consider the following examples.

Traffic from a restricted country vs pay per crawl

You may have both of the following features enabled:

Since WAF custom rules are enforced before pay per crawl, traffic (including AI crawlers) from your blocked countries will continue to be blocked, even if they provide the required headers for pay per crawl.

Allowed search engine bots via WAF custom rule vs pay per crawl

You may have both of the following features enabled:

Since WAF custom rules are enforced before pay per crawl:

  • Only search engine bots will be able to access your site (enforced by WAF custom rule).
  • The search engine bots will then be charged for access to your content (enforced by AI Audit's pay per crawl).

Conflict in AI crawler blocking logic

You may have both of the following features enabled:

  • A WAF custom rule which blocks all bots.
  • AI Audit selection which allows certain AI crawlers.

In this scenario, you have two WAF custom rules, each directing a different logic for handling AI crawlers. To resolve this issue:

  1. Log in to the Cloudflare dashboard, and select your account and domain.
  2. Go to Rules > Overview.
  3. Identify your WAF custom rule and the AI Audit rule.
  4. Drag the rule you wish to prioritize to the top, or modify your WAF custom rule to ensure it does not conflict with your AI Audit configurations.