Robots.txt Generator
Create a valid robots.txt for crawlers.
User-agent: * Disallow: /admin Sitemap: https://example.com/sitemap.xml
How to use
- 1 Add a user-agent rule (use * for all crawlers).
- 2 List the paths to allow and disallow.
- 3 Add your sitemap URL and an optional crawl delay.
- 4 Copy or download the robots.txt file.
About Robots.txt Generator
The Robots.txt Generator helps you write the small but important file that tells search engine crawlers which parts of your site they may and may not visit.
Add a rule for each user-agent — a catch-all asterisk for every bot, or specific agents like Googlebot or Bingbot — and list the paths to allow and disallow, one per line.
You can point crawlers to your XML sitemap and set an optional crawl-delay to ask well-behaved bots to slow down on a busy server.
The tool always keeps the output structurally valid, falling back to a sensible permissive default if you leave a rule empty.
A correct robots.txt keeps private or duplicate areas (admin panels, search result pages, staging paths) out of the index, conserves crawl budget on large sites, and surfaces your sitemap so new pages are discovered faster.
Remember that robots.txt is a request, not a security boundary: it discourages compliant crawlers but does not block access.
Everything is generated locally as you type, so you can copy the result or download a robots.txt file ready for your site root.
FAQ
Does robots.txt hide a page from everyone?
No. It only asks compliant crawlers not to fetch a path; the file is public and does not enforce access control. Use authentication for real protection.
Where should robots.txt live?
At the root of your domain, e.g. https://example.com/robots.txt. Crawlers only look for it there.