robots.txt Generator

Overview

Build a robots.txt file from user-agent groups, each with Allow and Disallow paths and an optional crawl-delay, plus a global Sitemap line. The file regenerates live as you edit the groups, with copy and download buttons for the finished text.

How It Works

The tool starts with two example groups: a wildcard group blocking /admin, /private, and /api, and a Googlebot group that allows everything with Allow: /. Edit each group's user-agent, Allow paths, and Disallow paths (one per line), and add a Crawl-delay (in seconds, written only when above 0) only if you want one. Add more groups with the button, or remove groups you do not need. Enter a sitemap URL at the bottom and the finished file is shown in the preview, ready to copy or download as robots.txt.

Step-by-Step Usage Guide

1. Edit the user-agent, Allow, and Disallow paths for the first group.
2. Add a group per crawler that needs different rules, using the add button.
3. Set the sitemap URL so crawlers can find your sitemap.
4. Copy or download the file and place it at the root of your domain.

Technical Specifications & Standards

The output follows the Robots Exclusion Protocol, which the IETF standardized as RFC 9309 in September 2022: groups start with User-agent, carry Allow and Disallow path lines, and the Sitemap line sits at the end where it applies globally rather than to any single group. Within each group this generator writes Allow lines before Disallow lines. For crawlers that follow RFC 9309, such as Google's, precedence between Allow and Disallow is decided by path specificity (longest match wins), so line order does not change the outcome; some older or simpler crawlers, however, are order-sensitive, which is why the generated order is stable and readable. Each group has a Crawl-delay input in seconds; a Crawl-delay line is written after the group's Allow and Disallow lines only when the value is a number above 0. Google ignores it, Bing and some other crawlers honor it, and crawlers that ignore it control crawl rate through their own webmaster tools instead, so do not rely on it for load management. Paths are trimmed and empty lines dropped but otherwise emitted verbatim with no validation, so wildcards and end-anchor dollar signs must be typed by you; an empty user-agent is written as *, and a group with no Allow or Disallow lines gets an explicit empty Disallow: line so it still allows everything. Nothing here enforces the file: robots.txt is a request, not access control, so keep real protections server-side.

Targeted Use Cases

• Blocking administrative and API paths from well-behaved crawlers on a new site.
• Giving a specific search engine different rules from the wildcard group.
• Publishing the location of your sitemap.xml alongside the crawl rules.
• Preparing a staging-to-production robots.txt handoff during a site launch.

Notes & Gotchas

• Never use robots.txt to hide private content; blocked paths are still reachable and the file is public.
• Type wildcards and the dollar end-anchor yourself where needed; paths pass through unvalidated.
• Test the generated file with a search console robots tester before deploying.
• Use absolute Sitemap URLs; the sitemap field is split on whitespace, so several URLs give several Sitemap lines, written once after all groups, and an empty field writes none.

Frequently Asked Questions