Skip to content
Cloudflare Docs

Manage AI crawlers

AI Audit enables you to take specific action for each AI crawler.

To manage AI crawlers:

  1. Log in to the Cloudflare dashboard, and select your account and domain.
  2. Go to AI Audit.
  3. Go to the AI Crawlers tab.

View the list of AI crawlers

The AI Crawlers tab displays a table of AI crawlers that are requesting access to your content, and how they interact with your pages. The table provides the following information.

ColumnDetails
AI CrawlersThe name of the AI crawler.
OperatorThe name of the entity who owns the AI crawler. Note that an operator may have multiple AI crawlers.
CategoryThe category of the AI crawler. Refer to Verified bot categories.
BlockToggle for blocking specific AI crawlers.

Take action for each AI crawler

For each AI crawler, you can take one of three actions: allow, charge, or block.

Allow access

You can allow an AI crawler to scrape your content.

From the Actions column, select Allow.

Note that you can still choose to Enforce robots.txt.

For more details on how this rule interacts with other Cloudflare settings, refer to How it works.

Charge for crawl (Enterprise only)

You can charge the owner of the AI crawler for each crawl request.

From the Actions column, select Charge.

For more information, refer to What is Pay Per Crawl?.

Block access

You can block an AI crawler to completely stop the AI crawler from scraping your webpage.

From the Actions column, select Block.

Filter AI crawler data

You can use filters to narrow the scope of your result.

  • Name: Search the name of the AI crawler.
  • Operator: Filter by the AI crawler operator.
  • Category: Filter by the category of the AI crawler (for example, AI crawler, AI assistant, or archiver).

The values of the table will update according to your filter.

  • Use pay per crawl to charge AI crawlers every time they access your content.