MundoKit

AI crawler robots.txt generator

Choose a policy for the listed AI agents, add optional path rules, and paste existing robots.txt to merge. Test a crawler token and path before downloading.

Create crawl preferences locally. robots.txt is voluntary and is not authentication.

How it works

Each policy writes a separate group for every listed agent. Training blocks GPTBot, ClaudeBot, Google-Extended and CCBot. Search and user-triggered groups remain allowed. Block all applies to the listed tokens only, not every AI service. Extra path rules apply to every listed agent.

The tester implements RFC 9309 sections 2.2.1–2.2.4: case-insensitive exact product-token groups combine, with wildcard groups used only if none match. Paths are case sensitive. The longest matching rule in octets wins; Allow wins equal lengths. It handles * wildcards, terminal $, UTF-8 percent encoding and unreserved encoded characters. Empty rules are ignored and /robots.txt is implicitly allowed. Enter a product token, not a full HTTP User-Agent header. No rounding is involved.

Merging replaces existing groups for listed tokens. In shared groups, other agents retain their rules. Unrelated text and Sitemap lines are retained; inspect the output before deployment. The tester ignores extensions such as Crawl-delay. Local limits are 100,000 bytes, 1,000 lines and 100 extra path rules; pattern evaluation has a bounded work budget.

Worked examples

Separate training from search

With the training policy and no extra rules, GPTBot is disallowed at /docs/ while OAI-SearchBot is allowed: 4 training tokens are blocked.

A more specific exception

With the block-all policy, add Allow: /public/. GPTBot can access /public/guide but cannot access /private/. The allow prefix is 8 octets, longer than the 1-octet root disallow.

Scope and operator differences

Google-Extended also controls grounding in certain Gemini products; it does not control Google Search inclusion. CCBot collects an open dataset with uses beyond training. OpenAI and Perplexity say user-triggered requests may bypass robots.txt. The tester describes RFC preferences, not actual enforcement. Robots.txt cannot secure private information.

Only verified tokens are included. Applebot-Extended, Meta-ExternalAgent and Bytespider are omitted because their operator documentation could not be verified in this review.

Frequently asked questions

Will this block every AI crawler?
No. It addresses the documented tokens listed in the tool. Crawlers can ignore robots.txt or spoof user agents. Use server access controls for private content.
Does allow all override my existing rules?
It replaces groups for the listed AI tokens with explicit Allow: / and your extra rules. Specific AI groups do not inherit User-agent: * rules. Other agents keep their existing groups.
Can I block training but retain AI search?
The training preset allows the listed search agents. Operator purposes differ: Google-Extended also governs some grounding, and CCBot is an open dataset crawler.
Does a longer Allow override Disallow: /?
Yes. Under RFC 9309, Allow: /public/ overrides Disallow: / for matching paths. Equal-length ties favor Allow.
How do I publish the result?
Review and serve it as UTF-8 plain text at /robots.txt on the relevant host. This tool does not deploy it or check server status, caching, redirects or crawler identity.

Sources

Last reviewed 2026-11-14; next review due by 2027-02-12. The method is written out above so you can check our math.