1
Symbiostock - SEO & Marketing / What is robots.txt for?
« on: September 07, 2013, 02:31 »
Search engine robots (bots) look at this file to see if they should crawl and index pages on your site, certain file types or even the entire site. An absence of this file gives them the green light to crawl and index your site