Search engine robots (bots) look at this file to see if they should crawl and index pages on your site, certain file types or even the entire site. An absence of this file gives them the green light to crawl and index your site

Traditionally its there to give your wishes of what you want crawled or not. Most bot's don't care or listen to it

But I understand bots like msn/google/yahoo mind it to a degree.

.