This site is for sale,
Learn More
Search Engines Indexing Problems
22 Reasons Why Your Page Did Not Get Indexed Part 2
Originally Published: November, 2003
Editor's Note:
Deep linked pages can be indexed by using a spider map. Use the
spider map creator
to create your free spider map.
More reasons why your site may not be indexed.
Non-Indexable Content:
It's important to know the types of content that the average search engine cannot index. Most engines cannot index text that is embedded in images. Text that appears in multimedia files (audio and video) will not be indexed. Most engines cannot index information that is generated by Java applets or in XML coding.
Large Pages:
If your site has a slow connection or the pages are very complex and take a long time to load, it might time out before the spider can index all the text. For the benefit of your visitors and the search engines, limit your page size to 50K or less. In fact, most Webmasters recommend that your page size PLUS the size of all your images on the page should not exceed 50K-70K total. If it does, many people on dial up connections will leave before the page fully loads.
Deep Links:
If you submit just your home page, don't expect a search engine to travel more than one or two links away from the home page or from the page that you submitted. Over time they may venture deeper into your site, but don't count on it happening quickly. You'll often need to submit pages individually that appear further down into your site or create more direct links from the home page. This way the search engines can find them. Visible links are preferable. If its not practical to provide a short link path from the home page to all your other Web pages, there are methods to create links that won't be seen by the average visitor.
The technique of submitting one page that then links to multiple other pages you want found is called creating a "hallway page" or a site map. In many cases, this will not only get you indexed in cases where they are ignoring you, but it will often improve your rankings. That's because many engines assign "bonus points" to pages they find on their own versus pages that were specifically submitted to them.
(
Editor's Note:
Create a
free search engine spider map
to make your site easy to index. )
Unreliable Hosts:
If your Web site fails to respond when the search engine spider pays a visit, you will not be indexed. Even worse, if you are indexed and they pay a visit when your site is down, you could be removed from their database! Consequently, it pays to have a reliable hosting service that is up 99.5% of the time. However, at some point a spider is going to hit that other 0.5% and end up yanking your pages by mistake. Therefore, it pays to keep a close eye on your listings and resubmit when needed.
SPAM:
If you have ever used any questionable techniques that might be considered an overt attempt at spamming (i.e., excessive repetition of keywords, same color text as background, or other things that the WebPosition Page Critic warns you about), an engine may ignore or reject your submissions. If you're having trouble getting indexed in the expected amount of time, make sure your site is spam-free.
Redirects:
If your site contains redirects or meta refresh tags, these things can sometimes cause the engines to have trouble indexing your site. Generally they will index the page that it is redirecting TO, but if it thinks you are trying to "trick" the engine by using "cloaking" or IP redirection technology that it can detect, there is a chance that it may not index the site at all.
(
Editor's Note:
Test 301 & other redirects here.
)
Proper Directory Submissions:
If you're submitting to a directory site like Yahoo, Open Directory, Looksmart, or others, then a human being will review your site. They must decide if the site is of sufficient "quality" before they will list it. I recommend you read the submission guide on the Guided Submission tab of the WebPosition Submitter. It contains tips on how to improve your chances of obtaining a good listing in these directories. Getting listed in major directories first can help you get listed elsewhere.
(
Editor's Note:
See our
search engine directories articles
for more information. )
Index Times Can Fluctuate:
WebPosition will tell you the average index time of each search engine. However, this is only an average. Sometimes engines will index sites every 30 days fairly consistently and then suddenly stop indexing most sites for several months. This can be frustrating, but it does happen. Generally a major engine will not go more than three to four months without refreshing its index.
Page Limits:
If you have many pages indexed but are having a hard time getting new ones recognized, be aware that there are limits. Each search engine will only spider so many pages of your Web site. This may range from a few dozen or three or four hundred depending on the engine. Some people have even been successful in getting far more pages indexed depending on the engine. Google is one engine that tends to crawl deeper into your site. However, how deep they go may depend on factors like your link popularity. Sites with higher link popularity are deemed "worthier" of more thorough indexing.
Random Errors:
Sometimes the engines just lose submissions at random through technical errors and bugs. After all, they are managing a database of hundreds of millions of pages. Therefore, some people like to resubmit once or twice a month for good merit in case they do lose a submission. Certainly if you've followed all the "rules" and are still not listed, by all means, re-submit! Sometimes a little persistence is all that's needed.
Link Farms:
If you've ever struggled to improve your link popularity, you may have been tempted by services offering instant fixes. Many such services are based on a network of sites that promise to link to you in exchange for you linking to them. While there's nothing wrong with exchanging links with other sites, you must be wary of such services that are considered to be spam by the major engines. These services are often termed "link farms."
Google in particular has been known to drop Web sites that participate in certain link exchange networks. If the links are being exchanged between unrelated sites for the sole purpose of increasing one's link popularity, then it's a good bet that Google and potentially other engines may choose to penalize all the sites participating in the network.
Therefore, avoid any kind of reciprocal link service where all members are required to link to a common Web site or script. If some of the text on the link page and/or the page name is the same among most or all linking parties, then watch out. It can become trivial for Google or another engine to identify all the sites in the linking service and to drop them from the index, or reduce their ranking. When reviewing any kind of link building service, be careful to note whether there's any way an engine could identify the members of the service.
If you've already joined one of these services and discovered your site dropped from the index, quickly remove any "tainted" link pages and then re-submit. If your page does not appear within the engines normal indexing time, then you may need to e-mail the engine and ask them to re-list you.
Indexing Problems Summary
If any of the above scenarios apply to your submission, you should take the appropriate actions and then re-submit. If that still does not work, you should consider e-mailing or calling the search engine and asking them politely why you have not been indexed yet. Sometimes they will reply back with "Sorry, there was a problem with our system and I've now made sure you'll be indexed within the next couple days." Or, sometimes they'll tell you why you were not indexed so you might correct it. In other cases, they will ignore your e-mail and you'll have to keep e-mailing or calling them until they respond. Still, it's definitely worth the effort to get your site listed with the major engines. This assumes you also take the time to optimize your pages so you'll achieve top rankings.
<<< Search Engines Indexing Problems, 22 Reasons Why Your Page Did Not Get Indexed Part 1
Editor's Note:
Errors in your robots.txt file can block spiders from your pages. Use our
robots.txt tester
to check your file.
This article is copyrighted and has been reprinted with permission from FirstPlace Software.
Site Promotion Articles Indexes:
|
