Robots File (robots.txt) Tester FAQ

Testing The robots.txt File Questions & Answers

Why Do I Need To Enter The Full URL Of My Robots File?

Entering the full URL allows you to test copy of a robots.txt file which allows you to correct any errors before installing it in the correct location.

What Errors Are Found With Your Tester?

The tester currently checks for typing errors (leading white space, etc) and structural errors (e.g. a disallow statement without a user agent statement). We are working on adding more tests and we welcome your suggestions made by contacting the webmaster.

Note that one error can hide another error, you should re-test your robots.txt file until no errors are reported.

I Don't Have A robots.txt File, How Can I Make One?

Use our Simple robots.txt Creator to make a robots.txt file for your site.

I Found An Error In My robots.txt Which Your Tester Didn't Catch

If you find an error which we missed please contact us and include a brief description of the error and a sample of the offending code.

Tester Terms & Conditions

We have tried to make this tester accurate but we can't guarantee it will catch all possible errors. Use of this tester is at your own risk and if in doubt you should consult the relevant standards.

Your robots.txt Tester Helped Me, How Can I Thank You?

We are glad the robots.txt tester helped you, please thank us with a link.

Where Can I Learn More About robots.txt Files?

Here are some useful sources of robots.txt information:

Make a robots.txt for your site: Simple robots.txt Creator
Practical robots.txt examples: Search Engine Spider Control With robots.txt
The robots.txt introduction: http://www.robotstxt.org/wc/norobots.html
The robots.txt RFC: http://www.robotstxt.org/wc/norobots-rfc.html

Return to the robots.txt checker

Get A Top 10
Search Engine Ranking