robots.txt Creator FAQ

Robots File Questions & Answers

Why Such A Simple Creator?

Most sites don't need to block individual robots so we decided to keep our creator simple by blocking all robots.

How Can I Test My robots.txt?

After installing your robots.txt file you can test it with our robots.txt tester

Security and robots.txt: Hiding Files & robots.txt

The robots.txt file has to be "world-readable" which means that anyone with a browser can read your robots.txt file looking for the location of sensitive files.

If you have files you want to keep hidden you should take them off of the server or at least protect them with passwords. This will also protect them from search engines which don't obey the robots.txt standard.

Where Can I Learn More About robots.txt Files?

Here are some useful sources of robots.txt information:

Practical robots.txt examples: Search Engine Spider Control With robots.txt
The robots.txt introduction: http://www.robotstxt.org/wc/norobots.html
The robots.txt RFC: http://www.robotstxt.org/wc/norobots-rfc.html

The robots.txt creator Helped Me, How Can I Thank You?

We are glad you found it useful, please thank us by linking to searchenginepromotionhelp.com

Return To robots.txt Creator

Get A Top 10
Search Engine Ranking