robots.txt Generator

Generate a valid robots.txt with allow and disallow rules, sitemaps, and AI crawler directives.

robots.txt Generator — robots.txt tells search engines and other crawlers which parts of your site they may visit. A well-formed file prevents wasted crawl budget and can block unwanted bots — including AI crawlers.

Your rules are generated locally in your browser. Nothing is sent to a server.

What is robots.txt generator?

robots.txt tells search engines and other crawlers which parts of your site they may visit. A well-formed file prevents wasted crawl budget and can block unwanted bots — including AI crawlers.

Pick which paths to allow and disallow, add your sitemap, and optionally include a section for AI crawlers. This generator produces a clean, standards-compliant robots.txt you can upload to your site root.

How to use it

Set global rules

Allow and disallow paths apply to all crawlers, e.g. /admin should be disallowed.
Add your sitemap

Include the full URL, for example https://yoursite.com/sitemap.xml.
Choose AI crawler rules

Optionally add a section that explicitly allows or blocks generative-AI crawlers like GPTBot and ClaudeBot.
Copy and upload

Save the output as robots.txt in the root of your domain.

Examples

A typical marketing site

Allow everything, point to the sitemap, and greet AI crawlers.

Allow: / · Sitemap: https://example.com/sitemap.xml

Result: A robots.txt with a universal allow rule, sitemap reference, and AI section.

Frequently asked questions

Where do I put robots.txt?

Should I block AI crawlers?

Is robots.txt a security measure?

Related tools

Sitemap Generator

Generate a clean XML sitemap for your website in seconds.

Meta Tag Generator

Generate title, meta description, canonical, Open Graph, and Twitter card tags as ready-to-paste HTML.

SERP Preview

Preview how your page title and meta description will look in Google search results before you publish.