Skip to content
Cloudflare Docs

Glossary

Review the definitions for terms used across Cloudflare's AI Audit documentation.

Term Definition
AI crawler

A bot which scrapes content from websites in support of an AI model, including by scraping content for indexing, retrieval augmented generation, or training.

Merchant of Record

The entity who facilitates "buying and selling". For pay per crawl, Cloudflare is the merchant of record.

robots.txt

A text file which lists pages in your website that are off-limits for bots. Well-behaved bots respect this file, but some bots may violate it. You can enforce robots.txt with Cloudflare WAF custom rules.