Robots.txt Generator
Create perfect robots.txt files instantly. Control search engine crawler access, add sitemaps, and configure crawl settings with our free visual builder.
Configure Your Robots.txt
🤖 Select Bots (Optional)
🚫 Disallow Paths
✅ Allow Paths (Exceptions)
🗺️ Sitemap URL
⏱️ Crawl Delay (seconds)
Generated Robots.txt
✓ ValidWhat is Robots.txt?
A robots.txt file is a simple text file that sits in your website's root directory and tells search engine crawlers (like Googlebot) which pages or sections of your site they can or cannot access. It's the first file crawlers check when visiting your site, making it a crucial component of your SEO strategy.
The robots.txt file follows the Robots Exclusion Protocol (REP), a standard used by websites to communicate with web crawlers. When properly configured, it helps you control how search engines crawl and index your content, preventing wasted crawl budget on unimportant pages and protecting sensitive areas of your site.
Why Do You Need a Robots.txt File?
- Control Crawl Budget: Prevent crawlers from wasting time on low-value pages like admin areas, thank-you pages, or duplicate content
- Prevent Indexing Issues: Block search engines from indexing private directories, staging sites, or development areas
- Improve Site Performance: Reduce server load by limiting crawler access to resource-intensive sections
- Manage Duplicate Content: Block crawlers from parameter-based URLs or printer-friendly versions of pages
- Sitemap Discovery: Include your sitemap URL to help search engines find and index your important pages
Basic Robots.txt Syntax
- User-agent: Specifies which crawler the rules apply to (e.g., User-agent: Googlebot or User-agent: * for all bots)
- Disallow: Paths that crawlers should NOT access (e.g., Disallow: /admin/)
- Allow: Exceptions to Disallow rules (e.g., Allow: /admin/public/)
- Sitemap: URL of your XML sitemap (e.g., Sitemap: https://example.com/sitemap.xml)
- Crawl-delay: Seconds between requests. Note: Google ignores this — use Google Search Console instead
Common Robots.txt Examples
Best Practices for Robots.txt
- Place in Root Directory: Always at https://example.com/robots.txt, never in subdirectories
- Test Before Deploying: Use Google's robots.txt Tester in Search Console to verify your rules
- Don't Block CSS/JS: Google needs these to render your pages for mobile-first indexing
- Not a Security Tool: Robots.txt is public — malicious bots can ignore it
- Case Sensitive: /Admin/ and /admin/ are different paths
- Include Sitemap: Always add your sitemap URL to help search engines discover your content
- Keep It Simple: Only block what truly shouldn't be crawled; over-blocking can hurt your SEO
Related Free SEO Tools
- XML Sitemap Generator — Create XML sitemaps
- Meta Tag Analyzer — Analyze meta robots tags
- .htaccess Generator — Server-level redirects and blocks
- Hreflang Generator — International SEO tags
- View All Free SEO Tools →
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file is a text file placed in your website's root directory that tells search engine crawlers which pages or sections of your site they can or cannot access. It's part of the Robots Exclusion Protocol (REP) and is the first file crawlers check when visiting your site.
Where should I place my robots.txt file?
The robots.txt file must be placed in the root directory of your website. For example, if your domain is example.com, the file should be accessible at https://example.com/robots.txt. It cannot be placed in subdirectories or subdomains.
What is the difference between Allow and Disallow?
Disallow tells crawlers NOT to access specific paths (e.g., Disallow: /admin/). Allow is used to make exceptions to Disallow rules, permitting access to specific paths within disallowed directories (e.g., Allow: /admin/public/).
Should I block my entire site with robots.txt?
No, unless your site is in development. Blocking your entire site (Disallow: /) prevents search engines from indexing your content, making your site invisible in search results. Only block specific sections like admin areas, private directories, or duplicate content.
What is crawl-delay in robots.txt?
Crawl-delay specifies the number of seconds crawlers should wait between successive requests to your server. Note: Google ignores crawl-delay — use Google Search Console to manage Googlebot's crawl rate instead.
Can robots.txt completely block bots?
No. Robots.txt is a suggestion, not a security measure. Well-behaved crawlers respect it, but malicious bots can ignore it. For actual security, use authentication, .htaccess rules, or server-level blocking.
How do I add a sitemap to robots.txt?
Add a line "Sitemap: https://example.com/sitemap.xml" to your robots.txt file. This helps search engines discover your sitemap. You can include multiple sitemap URLs if needed.
Is this robots.txt generator free?
Yes, 100% free with unlimited usage, no signup required, and no hidden costs. Generate as many robots.txt files as you need.
Fix Your Crawl Issues for Good
I've configured robots.txt, crawl budgets, and technical SEO for 2,000+ websites. Let me audit your entire crawl setup — robots.txt, sitemap, indexation — and make sure every important page gets found and ranked.