SEO Tips Blog
[ Blog Home ] Is robots.txt file a must?
Wednesday, August 23, 2006
Search engine crawlers first look into this special file when it visit your site and this text file tells which files should be indexed and which pages should be ignored in their search results.
Although this file is very important, use it only if you want to restrict some of your files being crawling by search engine robots. Here are some situations that could open a way to use this file on your site:
1) You want to restrict some robots crawling from entire website.
2) You have some directories contain private data that don't need to be indexed
3) Certain pages are temporary or not completed so don't need to index those till its completion.
4) You need to protect some files from viewing publically.
How to create robots.txt file
Robots.txt is a simple text file that can be created in Notepad. It should be saved to the root directory of your site. It is also easy to create a basic robots.txt file but if you misconfigure it, you can accidentally prevent your site from being spidered and indexed.
Following keys are allowed in the robots.txt file:
1) User-agent: Here you can specify the name of robot
2) Disallow: It tells the robot(s) which parts of your website are out-of-bounds.
Examples
To exclude a file named "sample.htm" located at the root of your site, from spidering Googlebot (Robot of Google):
User-Agent: Googlebot
Disallow: /sample.htm
To exclude a folder named "data" located at your root directory and its all files from all spiders and bots:
User-Agent: *
Disallow: /data/
To prevent all robots indexing your site:
User-agent: *
Disallow: /
Some useful links on robots.txt file:
Database of Web Robots
Robots.txt Syntax Checker
Robots META Tag
Robots.txt generator
[ Bookmark to del.icio.us ]
No comments yet.
 Free SEO Book
101 Essential Tips & SEO Tools
101 essential tips on search engine optimization, seo tools for analyzing your website and search engine / directory links.
Download free SEO ebook
Send this page to a friend |
Latest Posts
Bookmark This Blog
XML
|
