Free Preview: “...probably the best newsletter on search engine promotion in the world.” This month's issue

Why Search Engine Spiders Can't Index Your Pages

HTML Problems Can Prevent Indexing

Originally Published: November 13, 2007

This is a common problem. Your web designer has created a beautiful page with nice graphics and great Flash animations. Unfortunately, it seems that search engines won't list your website no matter what you do.

This can have several reasons:

a) The HTML Code Of Your Web Pages Is Meaningless To Search Engines

Search engines use very simple software programs to index your web pages. A web page that looks great to the human eye can be totally meaningless to search engines.

Search engines cannot see content that is presented in images (GIF, JPEG, PNG, etc.), Flash elements, JavaScript and other script languages or other multimedia file formats.

If you use JavaScript links for your website navigation then search engines might not be able to find your website page.

Solution: Check your web pages with a search engine spider simulator. Spider simulators will show you how search engines see your web site.

If you use IBP's search engine spider simulator, you can even enter the name of the spider, so that you can test if Google and Yahoo get different results when they visit your web pages.

b) The HTML Code Of Your Web Page Contains Errors

Some HTML errors can prevent search engine spiders from indexing your web pages. While most search engine spiders can deal with minor code errors, some of them will send faulty information to search engine spiders.

For example, your web page could contain a tag at the top of the page that tells search engines "the web page ends here" although your main content has not been reached.

Solution: Check the HTML code of your web pages with an HTML validator tool. You can find an HTML validator in the free IBP trial version.

c) The HTML Code Of Your Web Pages Doesn't Contain The Right Elements

If you want to get high rankings for a special keyword then this keyword must appear in the right places on your web page. For example, it usually helps to use the keyword in the web page title.

There are many other elements that are important if you want to have high rankings.

Solution: Check the HTML code of your web page with IBP's Top 10 Optimizer. IBP's Top 10 Optimizer compares your web page with the top ranked pages and it will tell you in detail how to change your web page so that it can get top 10 rankings.

d) Your Web Server Sends The Wrong Status Codes

Some web servers send wrong status codes. When a search engine spider requests a web page from your site then your server sends a response code. This should be the "200 OK" code.

Unfortunately, some servers send a "302 moved" or even a "404 not found" response code to the search engine spiders although the web page can be displayed in a normal web browser.

In that case, search engines will think that the web page doesn't exist and they won't index the page.

Solution: Use the search engine spider simulator to find out which response code your web server returns to search engines. If the response code is not "200 OK", IBP's search engine spider simulator will return a warning message. ( Editor's Note: You can also check your status code with our free HTTP Server Response Code Checker )

If you want to get top rankings on Google and other search engines then you must make sure that search engines don't misunderstand your pages. Use the tips above to make sure that search engines see what you want them to see.

Copyright by Axandra.com, publishers of:

• IBP All In One Promotion Software
  All You Need To Get Top 10 Rankings On Google And Yahoo!
• Successful Advertising With Google's AdWords
  How To Optimize Your Pay Per Click Campaigns
• Marketing ROI Optimizer
  Lower Your Costs And Increase Your Profits

All product names, copyrights and trademarks mentioned in this newsletter are owned by their respective trademark and copyright holders.

Site Promotion Articles Indexes:

Search Engine Promotion Help Site Map

©2008 Search Engine Promotion Help
May 09 2008 23:59 UTC