MicrostockGroup

Microstock Photography Forum - General => Symbiostock => Symbiostock - SEO & Marketing => Topic started by: aliceria on September 07, 2013, 02:31

Title: What is robots.txt for?
Post by: aliceria on September 07, 2013, 02:31

Search engine robots (bots) look at this file to see if they should crawl and index pages on your site, certain file types or even the entire site. An absence of this file gives them the green light to crawl and index your site

Title: Re: What is robots.txt for?
Post by: Leo Blanchette on September 07, 2013, 02:41

Traditionally its there to give your wishes of what you want crawled or not. Most bot's don't care or listen to it :D

But I understand bots like msn/google/yahoo mind it to a degree.

Title: Re: What is robots.txt for?
Post by: cathyslife on September 07, 2013, 06:34

.