This site is for sale, Learn More

How Your Firewall Can Destroy Your Search Engine Rankings

SonicWALL Firewall Blocks robots.txt File

Originally Published: May 24, 2005

Some webmasters recently experienced a delisting from the search engines without any good reason. The webmasters had not done anything wrong and their web sites were optimized for search engines. Nevertheless, the web sites had been removed from search engines.

Poorly configured firewalls can block search engine spiders

It turned out that the delisted web sites were all hosted by the same hosting company. More precisely, the web sites were all hosted by a hosting company that used a special firewall software by SonicWALL Inc.

That firewall stopped the search engine spiders from accessing their web sites. Google, Yahoo, MSN and all other search engines that request the robots.txt file couldn't index the web site anymore because the firewall didn't allow that:

"An attacker could retrieve robots.txt from the server, then use the contents of this file to discover the path of an unprotected administration interface for the server. The attacker may gain control of the webserver using this interface.

The information gathered from robots.txt could be used for system compromise and control of the web server." (source)

This is the standard security settings of the SonicWALL firewall and it basically means that your web site won't be spidered by search engines if you use this firewall without customizing it.

A firewall with these settings will drop the connection to anyone requesting the robots.txt file so that it looks as if the web site is offline. From an SEO point of view, this is very bad for your web site because all good search engine spiders request the robots.txt file before indexing your web site.

What does this mean to you?

If your web site is not listed on search engines although it has many good incoming links and optimized web page content, you should ask your web host if their firewall blocks search engines that request the robots.txt file. Your web host might not be aware of the problem.

Further information:

• How to create a robots.txt file for search engines
• robots.txt Creator, Generate Simple robots.txt Files
• robots.txt Checker, Test Your Robot File Syntax
• How to build good incoming links
• How to optimize your web site for Google, Yahoo, MSN Search and others

Copyright by Axandra GmbH, publishers of SEOProfiler, a complete SEO software solution.

Try SEOProfiler for free.

All product names, copyrights and trademarks mentioned in this newsletter are owned by their respective trademark and copyright holders.

Site Promotion Articles Indexes:

Search Engine Promotion Help Site Map

© 2024 Search Engine Promotion Help
November 23 2024 07:17 UTC