How to Use the Robots.txt Generator
1
Select a Preset or User-Agent
Choose a starter preset (Standard, Allow All, Next.js, WordPress) or pick a specific crawler.
2
Add Allow & Disallow Rules
Define path rules (e.g. Disallow: /admin/ or Allow: /public/).
3
Include Sitemap Location
Specify your full sitemap URL (e.g. https://domain.com/sitemap.xml).
4
Download & Deploy
Download robots.txt and upload it directly into your website public root directory.
Robots.txt Standard Syntax & Directives
The Robots Exclusion Protocol (RFC 9309) uses key directives:
| Directive | Syntax Example | Description | Bot Support |
|---|---|---|---|
| User-agent | User-agent: Googlebot | Identifies which spider the following rules apply to | All search engines |
| Disallow | Disallow: /admin/ | Prevents bots from crawling URLs starting with this path | All search engines |
| Allow | Allow: /public/ | Explicitly permits crawling within a disallowed parent directory | Google, Bing, Yandex |
| Sitemap | Sitemap: https://site.com/sitemap.xml | Declares sitemap location for automatic discovery | Google, Bing, DuckDuckGo |
| Crawl-delay | Crawl-delay: 5 | Requests a delay in seconds between consecutive crawler requests | Bing, Yahoo, Yandex |
Why Choose Our Robots.txt Generator?
Pre-configured templates for Next.js, WordPress, and standard websites
Instant toggle to block AI data scrapers (GPTBot, CCBot, Google-Extended)
Multi-rule builder with flexible Add / Remove path controls
Crawl-delay support for server load protection
One-click file download and copy to clipboard
Pro Tips for Best Results
- Robots.txt must always be placed at the root level: https://example.com/robots.txt.
- Disallowing a page in robots.txt does not prevent indexing if other sites link to it. Use <meta name="robots" content="noindex"> for complete de-indexing.