Robots.txt Generator

★★★★★★★★★★4.8(0 votes)

Generate a clean robots.txt with custom rules, AI-bot blockers, sitemap declaration, and popular presets.

👁 8 views❤ 0 likes⭐ 0 ratings💎 Free Tool

Robots.txt Generator

Default policy

Sitemap URL (optional)

Disallow paths (one per line)

Allow paths (overrides disallow)

Crawl-delay (seconds, optional)

robots.txt

User-agent: * Allow: /

Rate This Tool

Your rating helps improve ranking, recommendations and quality score.

4.8/50 users rated this tool

★★★★★★★★★★

Click a star to submit your rating

About This Tool

What is robots.txt?

The robots.txt file lives at yoursite.com/robots.txt and tells crawlers which parts of your site they may or may not access.

Common Use Cases

Block crawlers from admin, login, or staging URLs
Prevent indexing of duplicate pages
Declare your XML sitemap URL
Block AI training scrapers

Important

robots.txt is a polite request — malicious crawlers ignore it. Use server authentication or noindex meta tags to truly block content.

Frequently Asked Questions

Where do I upload robots.txt?

Place it at your domain root: yoursite.com/robots.txt. On WordPress, most SEO plugins (Yoast, Rank Math) let you edit it from the dashboard without FTP.

Will blocking AI scrapers actually work?

Major companies (OpenAI, Anthropic, Google) honor robots.txt. Less reputable scrapers may ignore it. For server-level enforcement, combine with IP blocking or Cloudflare WAF rules.

Should I block /wp-admin?

Yes, but allow /wp-admin/admin-ajax.php — many WordPress features depend on it. Our default templates handle this correctly.