Creating Robots.txt File
Frequently Asked Questions

Why do I need a robots.txt file?

Creating a robots.txt file will not improve your search engine positioning, but it does provide robots with information concerning which files you will not allow to be crawled and indexed in the search engines.

When a robot crawls your site it looks for the robots.txt file. If it doesn't find one it assumes automatically that it may crawl and index the entire site. Not having a robots.txt file can also create unnecessary 404 errors in your server logs, making it more difficult to track "real" 404 errors.

Assuming you want your entire site indexed and only want to stop the unnecessary 404 errors from occurring you have a couple of options.

Upload a blank robots.txt file to the root directory of your domain.
Upload a simple robots.txt file to the root directory of your domain.

What is a simple robots.txt file?

This allows all robots to crawl all files.

User-Agent: *
Disallow:

What if I don't want a particular file crawled?

Please note: Disallowing a file to be crawled will keep it from being indexed. The file disallowed will not show up in the search engines.

This allows all robots to crawl all files except the images file.

User-agent: *
Disallow: /images/

This allows all robots to crawl all files except the images file and the stats file.

User-agent: *
Disallow: /images/
Disallow: /stats/

What if I want to disallow a particular robot?

Occasionally you may find that you would like to disallow specific robots from crawling your site or limit which files they may have access to.

This denies access to Googlebot-image to any files in your domain

User-agent: Googlebot-Image
Disallow: /

This specifically denies Googlebot-image to your images file

User-agent: Googlebot-Image
Disallow: /images/

For a current data base of robot names and information, visit: http://www.robotstxt.org/wc/active/html/index.html

How do I create a robots.txt file?

Simply create a text document and save the new document as robots.txt

Do not use a html editor to create the file unless is has the ability to create a plain text document (ASCII).

Most computers will allow you to create a text document using notepad.

Right click on your desktop
Choose new
Choose text document
Open the document you just created
Insert instructions to robots
Click on save as
Save document as robots.txt

How do I know if I have done everything correctly?

Once you have uploaded the file to the root directory of your domain it's good idea to use a robots.txt validator to confirm that everything is correct. Search Engine World provides a FREE robots.txt vaidator.
http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

What if I need more information about robots.txt files?

This page is intended to cover creating a very simple robots.txt file. If you require a more detailed robots.txt file for your website there are many help resources available on the net. Google Information for Webmasters recommends visiting: http://www.robotstxt.org/wc/norobots.html

Sometimes a program is so incredible that we have to share it with others! We are extremely impressed with this all new version of Xara Webstyle 4 and are currently using it as a tool in designing our sites! This is a "must have" tool for web designers! Take a look at some of the new features and download the FREE TRIAL TODAY!

Create Complete Themed Web Page Layouts
Integration with Dreamweaver and FrontPage
Design Navigation Bars and Menus
Create Complete Web Photo Albums
Design Banners, Logos, Buttons and Bars

Try Xara Webstyle 4 FREE Today!

Creating Robots.txt File Frequently Asked Questions