AI Audit with Cloudflare WAF

AI Audit works alongside other Cloudflare products, such as Cloudflare Web Application Firewall (WAF). WAF checks incoming web and API requests, and filters undesired traffic based on rules. WAF custom rules allow you to perform certain actions such as enforcing robots.txt.

Order of precedence

• AI Audit uses WAF custom rules to block the selection of AI crawlers the site owner has decided to block.
• AI Audit's pay per crawl feature takes place after WAF.

graph LR
A[Traffic] --> B[WAF custom rules<br>AI Audit: Crawler blocks]
B --> C[Cloudflare<br>Bot Solutions]
C --> D[AI Audit:<br>Pay Per Crawl]
classDef highlight fill:#F6821F,color:white

For this reason, if you plan on using AI Audit to manage AI crawlers, you may wish to modify your existing WAF custom rules such that it does not affect AI crawlers. This will allow you to manage AI crawlers only from AI Audit, thereby streamlining your workflow.

How AI Audit uses WAF custom rules

When you block AI crawlers via AI Audit (either all or some), you are using one WAF custom rule to block those AI crawlers.

If you choose to allow all AI crawlers, AI Audit does not utilize any WAF custom rules.

Depending on the type of account you have, you may have a limited number of WAF custom rules.

Examples of using WAF vs AI Audit

Consider the following examples.

Traffic from a restricted country vs pay per crawl

You may have both of the following features enabled:

• WAF custom rule to block traffic from specific countries
• AI Audit's pay per crawl to charge AI crawlers when they request access to your content

Since WAF custom rules are enforced before pay per crawl, traffic (including AI crawlers) from your blocked countries will continue to be blocked, even if they provide the required headers for pay per crawl.

Allowed search engine bots via WAF custom rule vs pay per crawl

You may have both of the following features enabled:

• WAF custom rule to allow search engine bots
• AI Audit's pay per crawl to charge all AI crawlers when they request access to your content (including search engine bots).

Since WAF custom rules are enforced before pay per crawl:

• Only search engine bots will be able to access your site (enforced by WAF custom rule).
• The search engine bots will then be charged for access to your content (enforced by AI Audit's pay per crawl).

Note

This example only serves to highlight the order of precedence between WAF and AI Audit.

Practically, it may be beneficial to allow well-behaved search engine bots to access your content to ensure your content is indexed.

Conflict in AI crawler blocking logic

You may have both of the following features enabled:

• A WAF custom rule which blocks all bots.
• AI Audit selection which allows certain AI crawlers.

In this scenario, you have two WAF custom rules, each directing a different logic for handling AI crawlers. To resolve this issue:

1. Log in to the Cloudflare dashboard ↗, and select your account and domain.
2. Go to Rules > Overview.
3. Identify your WAF custom rule and the AI Audit rule.
4. Drag the rule you wish to prioritize to the top, or modify your WAF custom rule to ensure it does not conflict with your AI Audit configurations.