This site is for sale,
Learn More
Search Engine Promotion Mistakes
Site Down, Bad robots.txt File
Continued From:
<<< Shared IP Address, Free Hosting
Many webmasters have the problem that their Web site is not listed in search engines at all. There can be a variety of reasons that your Web site doesn't show up on search engines.
#11: Your Host Server Was Non-Operational During Spidering
It can happen that your Web server is down when a search engine spider tries to index it.
If your Web site fails to respond when the search engine spider visits your site, your site will not be indexed. Even worse, if your Web site is already indexed and the search engine spider finds that your site is down, you'll often be removed from the search engine database.
It's essential to have Web space on servers that are very seldom down. Choosing a reliable Web space provider is very important for a successful online business.
At first it sounds impressive when your Web space provider promises 99% server reliability. But take a moment to calculate it. It means that 1% of the time, your potential customers cannot reach your Web site. One percent of a year means that your Web site will be down for nearly 4 days per year. That equals 4 days without sales.
As you can see, 99% reliability is not enough. You must constantly monitor your server so that you can act immediately if your server is down (e.g. call the service provider to restart the server).
Here's a free monitoring service:
http://www.internetseer.com
Another free Web site monitoring service (without ads):
http://uptime.openacs.org/uptime/
A freeware tool that can monitor the uptime of your site:
http://www.idyle.com/software/internet/host-monitor/
Monitoring service for bigger companies:
http://www.atwatch.com/
#12: You Don't Allow Robots To Index Your Web Site
Imagine you're a Internet marketing service company and you keep trying very hard to get a top ranking in the search engines for your customer.
Even after several weeks, the customer's Web site hasn't been listed in any search engine. Then you start to realize that the search engine spiders and robot programs cannot access the Web site because your customer blocks them (by mistake).
There are two ways to block search engine robots: a) with a simple text file in the root directory of the host server, or b) with a certain META tag in the Web pages.
a) Robots.txt
The host server might have a plain text file named "robots.txt" in the root directory. It contains rules for the search engine spiders. The rules in the robots.txt file follow the Robots Exclusion Protocol, a document designed to help Web administrators and authors of Web spiders agree on a way to navigate and catalog Web sites.
The content of the robots.txt file consists of two main commands: "User-agent" and "Disallow".
The User-agent command specifies the name of the robot for which the following commands should be applied to. You can set this to "*" to have the spidering commands applied to any robot.
The second command, "Disallow", specifies a partial URL that should not be indexed by the Web robot.
The text
---
User-agent: *
Disallow: /
---
tells all search engine spider programs to go away. If you find a text file called "robots.txt" in the root directory of the host server with the above content, you should delete it immediately. The text file says that no search engine is allowed to index your Web site.
Even if your robots.txt file don't contain the above commands, you should make sure that its syntax is correct. A robots.txt file with a faulty syntax also prevents search engine spiders to index your Web site.
(
Editor's Note:
Use the
robots.txt tester
to check your robots file for errors or create a new one from scratch using the
Simple robots.txt Creator.
)
b) The META ROBOTS tag
There's a second way to stop search engine robot programs to index your Web site: the META ROBOTS tag. If you find the following HTML tag in your Web pages:
---
<META NAME="robots" CONTENT="noindex,nofollow">
---
you should replace it immediately with
---
<META NAME="robots" CONTENT"="index,follow">
---
If you want all search engine spiders to index all Web pages, you can also remove the META ROBOTS tag from your Web pages.
Further information about both ways to stop search engines to index your Web site can be found at:
robots-txt-explained.php
- http://www.wdvl.com/Location/Search/Robots.html
- http://www.ebrandmanagement.com/whitepapers/robots2.htm
Continued:
Full-Fledged Browser Required, Bad DNS Settings >>>
Continued From:
<<< Shared IP Address, Free Hosting
Copyright by Axandra GmbH, publishers of SEOProfiler, a
complete SEO software solution.
Try SEOProfiler
for free.
All product names, copyrights and trademarks mentioned in this newsletter are owned by their
respective trademark and copyright holders.
Site Promotion Articles Indexes:
|
