Free Preview: “...probably the best newsletter on search engine promotion in the world.” This month's issue

robots.txt Creator, Generate Simple Robots Files

Enter The Files And Folders To Protect:

Enter each file name or folder path which you want to protect from all robots. This path is the path from the root of your server. If a file has a URL like www.your-domain.com/pages/hide-me/hidden.html then the path is /pages/hide-me/hidden.html

Sample file entry: "/logs/log-referr.html"

Sample folder entry (end a folder name with a "/" character): "/cgi-bin/"

Caution: If you enter just a "/" character this will block all spiders from indexing all files on your site!

Adding A Sitemap Tag To Your robots.txt File:

The last field allows you to enter the URL of your Sitemap if you have one on your site. This should be a complete URL like http://www.searchenginepromotionhelp.com/_sitemap-dynamic.xml.gz.

If you don't have a sitemap, generate your sitemap with our free online form.

robots.txt creator FAQ & Additional information

We hope you found this robots.txt creator helpful, please consider saying "thank you" with a link

We welcome your comments about this creator, please contact the webmaster.

Search Engine Promotion Help Site Map

©2008 Search Engine Promotion Help
May 11 2008 23:21 UTC