pancakes

MicrostockGroup Sponsors


« previous next »
Home | Pages: [1]

Author Topic: What is robots.txt for?  (Read 4147 times)

0 Members and 1 Guest are viewing this topic.

aliceria

« on: September 07, 2013, 02:31 »
0
Search engine robots (bots) look at this file to see if they should crawl and index pages on your site, certain file types or even the entire site. An absence of this file gives them the green light to crawl and index your site
Logged

Leo Blanchette

« Reply #1 on: September 07, 2013, 02:41 »
0
Traditionally its there to give your wishes of what you want crawled or not. Most bot's don't care or listen to it :D

But I understand bots like msn/google/yahoo mind it to a degree.
Logged

cathyslife

« Reply #2 on: September 07, 2013, 06:34 »
0
.
Logged
Home | Pages: [1]
« previous next »
 

Related Topics

  Subject / Started by Replies Last post
5 Replies
2481 Views 		Last post September 03, 2013, 15:46
by cathyslife

Sponsors

Mega Bundle of 5,900+ Professional Lightroom Presets

Microstock Poll Results

Sponsors