Robots.txt is what a search engine spider looks for when crawling your site. It tells the spider which pages on your website to index and which to leave alone meaning this small file is of large importance. It is a simple text file (no HTML) to be placed in your root directory like this:
http://www.analyticsseo.com/robots.txt
It’s important as you can ensure search engine spiders spend what little time they have allocated to your website indexing the right content.