Robots TXT Generator
Build crawler rules from a form, explain each rule, and preview the file before download.
One path per line, starting with /. A path ending in / covers everything inside that folder.
Your result
Your robots.txt appears here.
Upload the file to the root of your site, so it opens at example.com/robots.txt. Robots.txt is a request, not a lock: it does not hide private pages.
How to use it
- Choose everything, everything except some paths, or nothing.
- Add paths to block with the quick buttons or by typing, plus your sitemap address.
- Download robots.txt and upload it to your site root.
Examples
- AllowAllow every path # Allow every path. User-agent: * Allow: /
- Block allBlock every path # Block every path. User-agent: * Disallow: /
- One pathBlock /admin # Block one path. Other paths are not mentioned. User-agent: * Disallow: /admin
Tool details and privacy
Choose what search engines may crawl, list paths to block, add your sitemap, and download a ready robots.txt.
Each choice writes a comment that says what the rule does, so the file stays readable later.
Options
- Each blocked path starts with /. Several paths are allowed, one per line.
- An option turns away the main AI training crawlers.
Limits
- The file is not tested on a live site.
- robots.txt is a request to crawlers, not a lock.
Privacy
What you type stays in your browser. Nothing is uploaded or saved on the server.
Questions
Does the page test the file on my site?
No. Use the robots.txt tester for that.
What does Nothing write?
Disallow: / for every crawler.
Does blocking a page hide it?
No. Anyone can still open it, and it may still appear in search if other sites link to it.
Is anything uploaded or saved on the server?
No. It stays in your browser.