Free tool
Decide which AI crawlers can read you
This builds a robots.txt that names every major AI crawler explicitly, so you decide who reads your pages instead of guessing. Defaults allow all of them, because blocking a crawler removes you from that assistant's answers entirely. Private paths repeat inside each group, which is how the file actually works.
Last reviewed 27 August 2026
Result
AI crawlers allowed
9 of 9
100% of the named tokens can read your pages
robots.txt
# robots.txt # AI crawlers are named explicitly so the decision is recorded, not inherited. # Every group repeats the private paths on purpose: a crawler that matches # its own group ignores the User-agent: * group entirely. User-agent: GPTBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /account/ User-agent: ChatGPT-User Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /account/ User-agent: OAI-SearchBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /account/ User-agent: ClaudeBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /account/ User-agent: PerplexityBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /account/ User-agent: Google-Extended Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /account/ User-agent: Applebot-Extended Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /account/ User-agent: CCBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /account/ User-agent: Bytespider Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /account/ User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /account/ Sitemap: https://acme.com/sitemap.xml
What each token does
| User agent | Operator | Setting | Effect |
|---|---|---|---|
| GPTBot | OpenAI | Allowed | Crawls pages to build training data |
| ChatGPT-User | OpenAI | Allowed | Fetches pages when a user asks, and builds the ChatGPT search index |
| OAI-SearchBot | OpenAI | Allowed | Fetches pages when a user asks, and builds the ChatGPT search index |
| ClaudeBot | Anthropic | Allowed | Crawls pages for Claude |
| PerplexityBot | Perplexity | Allowed | Crawls pages for Perplexity answers and their citation links |
| Google-Extended | Allowed | Separate control for Gemini and AI features, per Google's documentation | |
| Applebot-Extended | Apple | Allowed | Separate control for Apple Intelligence, not for Siri or Spotlight indexing |
| CCBot | Common Crawl | Allowed | Bulk collection that feeds many downstream models and datasets |
| Bytespider | ByteDance | Allowed | Bulk collection that feeds many downstream models and datasets |
Before you publish
- Publish at the root of the domain. A robots.txt at /docs/robots.txt does nothing, and rules do not carry across subdomains.
- Serve it as text/plain and check it returns 200. A soft 404 page is read as an empty file, which allows everything.
Everything is allowed, which is the right default for most companies. robots.txt is a request rather than an enforcement mechanism: it is honoured by the crawlers that choose to honour it and ignored by the rest, so never use it to protect anything that actually needs protecting.
How this works
What the numbers mean.
- 01Each crawler gets its own group with its real user-agent token. Grouping matters: robots.txt parsers apply the single most specific matching group, so a bot with its own group never reads the rules under User-agent: star. That is why the private paths are repeated inside every group here rather than stated once.
- 02Allowed crawlers get Allow: / followed by your private path disallows. Blocked crawlers get Disallow: / and nothing else, which is unambiguous across parsers.
- 03The Sitemap directive is independent of any group, so it sits at the end and applies to every crawler that reads the file. It is only emitted if you give an absolute URL.
Assumptions and limits
- robots.txt is voluntary. Well-behaved crawlers respect it and badly behaved ones do not, so it is a preference signal rather than access control.
- Blocking a training crawler and blocking a retrieval crawler are different decisions with different costs. Retrieval crawlers are the ones that decide whether an assistant can cite you today.
- This file cannot remove content already in a trained model, and it does not affect what other sites say about you, which is where a lot of AI answers actually come from.
Questions about this tool
Should I block AI crawlers?
Usually not. If you sell software, services, or anything where being recommended matters, blocking removes you from the answers your buyers are reading and gives the citation to a competitor who stayed open. The case for blocking is strongest when the content itself is the product, such as a paid publication or a research archive, and weakest for everyone else.
What is the difference between GPTBot and ChatGPT-User?
OpenAI documents GPTBot as the crawler that gathers content for training, while ChatGPT-User fetches a page in response to a live user request and OAI-SearchBot builds the search index behind ChatGPT's results. The distinction matters commercially: blocking GPTBot is a data-rights position, while blocking the other two takes you out of answers being generated right now.
Does Google-Extended affect my Google Search ranking?
Google documents Google-Extended as a separate control for Gemini and its AI features, not for Search indexing or ranking. Blocking it is a decision about generative uses of your content specifically. Confirm against Google's current crawler documentation before you act on it, because these controls have changed more than once.
Why do the private paths repeat in every group?
Because of how robots.txt matching works. A crawler picks the single most specific group that matches its user-agent and ignores everything else, including the catch-all group. If your disallows only appear under User-agent: star, every bot you named by token is free to crawl those paths. Repeating the rules is the only way to make them apply.
This tool is free and there is nothing to sign up for. If you would rather have the work done than calculate it, that is what Zway does.