Search Engines Indexing Problems
22 Reasons Why Your Page Did Not Get Indexed Part 1
Originally Published: November, 2003
Editor's Note:
This article discusses general search engine indexing problems. For more information about search engine spidering problems see
Search Engine Spiders Miss Pages
If there's one question in search engine marketing that comes up far too often it is: "Why hasn't my page or Web site been indexed yet?" Or, almost as often: "Why was my page indexed for a while and has now disappeared?"
Since MarketPosition's first issue in September 1997, I've rarely, if ever, repeated an article. However, the article addressing this important question is a classic. It provides a comprehensive review of the many pitfalls in search engine submission. The original article nearly two years ago included 21 reasons for not being indexed. This latest revision includes today's top 22 reasons. Even SEO veterans will find something of value here.
At one time or another you may have used a submission tool, or submitted by hand and then wondered why you had not been indexed. Unfortunately, there are many reasons that may delay or prevent you from being indexed by a search engine. There's rarely one simple answer for why you're not being found. Fortunately, there is usually an explanation and a way to correct the problem if you know what to look for.
Below are the Top 22 reasons we've compiled over the years as to why you may not be finding your Web site or Web page in one or more search engines:
Index Time:
First, make sure you've allowed enough time to become indexed. The amount of time to allow is sometimes listed on the search engine's submission page. Unfortunately, the engine's own advertised times are often inaccurate or out of date.
WebPosition's Submitter report and WebPosition's URL Verification report will both tell you how much time you should allow for each engine before being concerned about not being indexed. Average index times often range from one to eight weeks depending on the engine. Some engines like AltaVista and Inktomi offer paid options if you wish to be indexed more quickly.
Already Indexed:
Be sure you're not already indexed but just don't know it. Unfortunately, none of the major engines are kind enough to e-mail or notify you as to if and when you've been indexed.
In addition, you cannot simply do a search on a keyword that applies to your Web site and expect it to pop up at the top. In fact, you must take pro-active steps to optimize your pages for each search engine. If you don't, it's very unlikely you'll find your Web site except on the most specific of searches.
The method to determine if a page or domain has been indexed varies from one engine to another, and in many cases, it's difficult to tell for sure whether your pages are in fact in there. Never assume that you're not indexed just because you searched for a bunch of keywords and you did not come up in the first few pages of results. You could be there (i.e., indexed) but be buried near the bottom.
In addition, it's not very practical to check the status of a number of pages on each major engine each week. Fortunately,
WebPosition Gold
has a URL verification feature in the Reporter that makes this process much easier. Each time you run a mission, it will report which URLs exist and do not exist in each engine.
Missing Page:
Make sure you have uploaded the pages to your site before submitting them. This one will seem obvious to many people, but submitting a page that does not exist or submitting with a subtle typo in the URL is a goof we might all make at one time or another. If you're using WebPosition's Submitter, there's a checkbox on tab 2 labeled "Verify that each page exists on Web site before submitting." This option defaults so that WebPosition will verify that all your URLs are valid and actually exist before submitting them. This is important since not all search engines will notify you if the URL does not exist when you submit.
Roadmap From Home Page:
Some engines have been known to drop pages that cannot be traveled to from the home page. HotBot has been rumored to do this. You may want to consider submitting your home page that links either directly or indirectly to your doorway pages. Think of your Web site as a series of roads (i.e., links) from one page to another. If there's no road from your home page to the page you want indexed, a search engine may decide the page is unimportant or of low-quality. You could submit the page directly, but the engine may reject it or may drop it at a later date when it finds no "road."
(
Editor's Note:
Create a
free search engine spider map
to make your site easy to index. Also, we can create a
Script To Create Sitemap From Database
)
External Links:
Some search engines such as Google and HotBot have been known to refuse to index Web sites that do not have any other Web sites linking to them. Or, they may index your home page but refuse to index any other pages until you achieve at least one or more links from another domain. Or, they may index you for a while but then "prune" their database later of all Web sites that did not achieve any external links within a certain period of time. However, do not worry! You simply need to establish some links and when that's done, resubmit both your pages and the pages that link to you. Once you have links to your Web site, it becomes much easier to get indexed, stay indexed, and to achieve top rankings.
(
Editor's Note:
See our
link popularity articles
for more information. )
Frames:
If you have content inside HTML frames, this can cause problems with submissions. For example, the search engine may index the main content of the page, but not the surrounding menu frame. Visitors to your site will then find some information but may not see the associated menu! It's generally best if you can create non-framed versions of your pages. You should then submit the non-frames versions of your pages, which can of course link to your framed Web site. Alternatively, you can enter your relevant text within the NOFRAMES area of a framed page that most search engine spiders will read. However, don't expect to achieve high rankings while optimizing the NOFRAMES area. Optimizing a NON-framed page will often achieve better results.
Spider Blocks:
Search engine spiders cannot index sites that require any kind of registration or password. A spider cannot fill out a form of any kind. The same rule applies regarding indexing of content from a searchable database. That's because the spider cannot fill out a form to query that database. The solution is to create static pages that the engines will be able to find and index without performing a special action on your site. Depending on the database system you have, there are utility programs out there that help you do this, as well as companies that can assist you.
Free Sites:
Many engines no longer index pages from free web sites or they limit the number of pages they will index from these hosts. Sometimes they will get too many "junk" submissions from free web site domains such as Geocities or others. Therefore, some engines choose not to index anyone for some of these domains. Or, more commonly, they limit the number of pages they will accept.
It's always best to buy your own domain name (very important) and place it on a respected, paid service to avoid being discriminated against. The free traffic you can generate from the search engines is just too valuable to be sacrificed for the small savings a free hosting service provides. In addition, free hosts are often unreliable or force you to display banners that send valuable visitors away from your Web site soon after arriving. That can cost you sales.
Guilt Through Association:
If your Web site shares the same IP address as many other Web sites on your host's Web server, then you may find your IP quietly banned from something another Web site on the same server did! It's always best to ask your hosting service if your domain name has its own unique IP assigned to it. If not, ask them to move it to its own IP to avoid the potential of having your submissions ignored because of something that a site sharing your IP did. We've heard from many people who tried everything to be indexed only to find it was a snap once they changed hosting services.
Submission Limits:
Make sure you're submitting within the recommended limits. Some engines do not like more than a certain number of submissions per day for the same domain. If you exceed the limit, you may find that all your submissions for that day are ignored. Fortunately, WebPosition's submitter will warn you regarding current limits and help keep you within them. Some submission consultants feel it is dangerous to submit more than ONE page a day to an engine for a given Web site. For those who wish to be ultra-conservative in their approach, the WebPosition Submitter includes a checkbox to limit submissions to one URL per day per engine.
Dynamic Pages:
Dynamic pages are often ignored by the search engine spiders. In fact, any URL containing special symbols like a question mark (?) or an ampersand (&) will be ignored by many engines. Pages generated on the fly from a database often contain these symbols. In this situation, it's important to generate "static" versions of each page you wish to be indexed. In regard to the search engines, the simpler the page is, the better. Does this mean, for example, having a JavaScript to count visits to the page will prevent you from being indexed, or lower your rankings? No. It simply means that the search engine will most likely ignore the JavaScript and index the remaining areas of the page. There is evidence that going too far with fancy scripts and code on a page can hurt your rankings if the bulk of your page consists of java or VB scripts.
Search Engines Indexing Problems, 22 Reasons Why Your Page Did Not Get Indexed Part 2 >>>
Editor's Note:
The sitemap protocol makes indexing dynamic sites easier. We now offer scripts to create sitemaps to this protocol from databases. Learn more here:
Script To Create Sitemap From Database
This article is copyrighted and has been reprinted with permission from FirstPlace Software, the makers of WebPosition.
FirstPlace Software helped define the SEO industry with the introduction of the first product to track your rankings on the major search engines and to help you improve those rankings.
Site Promotion Articles Indexes:
|
