Question 1

Where do I upload the generated robots.txt?

Accepted Answer

To your site's root directory so it's reachable at https://yourdomain.com/robots.txt — it must be at the root, not in a subfolder, for crawlers to find it.

Question 2

How do I block AI crawlers but keep Google search working?

Accepted Answer

Use the "Block AI crawlers, allow search engines" preset — it adds a Disallow: / rule for each major AI crawler (GPTBot, CCBot, ClaudeBot, and others) as its own separate user-agent group, while leaving the User-agent: * group open, since Googlebot and Bingbot fall under the wildcard, not the AI-specific groups.

Question 3

Does robots.txt actually stop AI companies from using my content?

Accepted Answer

It signals your preference to crawlers that choose to respect it — reputable crawlers from major AI companies do honor robots.txt Disallow rules for their listed user-agents, but robots.txt is a voluntary convention, not an enforcement mechanism.

Robots.txt Generator

Frequently Asked Questions

More Free Tools