AI Crawler Policy Generator
Use this tool to declare how AI crawlers — ChatGPT, Claude, Perplexity, and others — are permitted to use your website's content. Choose your policy, select which bots to target, and get ready-to-use output for your robots.txt file and your server's HTTP response headers — both based on Cloudflare's Content Signals specification.
Google has publicly stated it will not honor the Content-Signal directive, citing its preference for existing robots.txt and meta tag standards. Compliance among other AI crawlers varies. This tool reflects the current Cloudflare spec — treat the output as a declaration of intent, not a technical enforcement mechanism. That said, establishing your position now matters as the standard evolves.
2. Target specific bots (optional — defaults to all)
3. Your generated directives
Add this block to your robots.txt file at the root of your domain (e.g., yoursite.com/robots.txt). WordPress users can edit it inside Yoast SEO → Tools → File Editor, via RankMath → General Settings → Edit robots.txt, or directly via FTP at /public_html/robots.txt. If you already have a User-Agent: * block, insert the Content-Signal: line inside that existing block rather than creating a duplicate.
This is a server-level HTTP response header that signals your content policy on every page request — not just when a crawler reads robots.txt.
Not sure which policy is right for your business? AI visibility strategy is part of what we do — and it's more nuanced than a single checkbox. Let's talk through it.
Talk to Blake →