Robots TXT Rule Tester
Paste a robots file and paths to test allow and disallow with a stated rule engine, without fetching the site.
Open yoursite.com/robots.txt, copy all of it, and paste it here.
One per line. A path such as /blog/post, or a full address.
Your result
Paste your robots.txt and the paths to test. Each path gets an answer here.
Read the way Google does: the most specific crawler group applies, the longest matching rule wins, Allow wins a tie, * matches any text, and $ marks the end. Other search engines may differ slightly.
How to use it
- Paste the robots rules.
- Type the paths.
- Read allow or disallow for each path.
Examples
- A blocked folderDisallow: /private/, path /private/a /private/a — Blocked by Disallow: /private/
- Allow wins a tieAllow: /a and Disallow: /a, path /a /a — Allowed
- A wildcardDisallow: /*.pdf$, path /files/list.pdf /files/list.pdf — Blocked
Tool details and privacy
Paste your robots.txt and the paths or addresses to test. Each one is marked Allowed or Blocked, with the rule that decided it.
A pasted file can be checked. It is not Google’s live tester.
Options
- Choose a crawler such as Googlebot; its own group is used when the file has one, otherwise the * group.
- Supports * and $ patterns; the longest matching rule wins and Allow wins a tie.
Limits
- The site is not fetched.
- This follows Google's rules; other crawlers may differ slightly.
Privacy
What you type stays in your browser. Nothing is uploaded or saved on the server.
Questions
Does this fetch my site?
No.
Which rule wins when two match?
The longer one. If they are the same length, Allow wins.
Can I test a full address?
Yes. Only its path and query are used.
Is the file uploaded?
No.