Welcome,
Guest
. Please
login
or
register
.
Did you miss your
activation email
?
1 Hour
1 Day
1 Week
1 Month
Forever
Login with username, password and session length
Home
Forum
Help
Tools
Keyword Research Tool
Keyword Comparison
Polls
History Graph
Submit Vote
Support
Advertise
Premium Membership Sign-Up
Search
Login
Register
MicrostockGroup
»
Microstock Photography Forum - General
»
Symbiostock
»
Symbiostock - SEO & Marketing
»
What is robots.txt for?
MicrostockGroup Sponsors
« previous
next »
Print
Home
| Pages: [
1
]
Author
Topic: What is robots.txt for? (Read 3462 times)
0 Members and 1 Guest are viewing this topic.
aliceria
«
on:
September 07, 2013, 02:31 »
0
Search engine robots (bots) look at this file to see if they should crawl and index pages on your site, certain file types or even the entire site. An absence of this file gives them the green light to crawl and index your site
Logged
Leo Blanchette
«
Reply #1 on:
September 07, 2013, 02:41 »
0
Traditionally its there to give your wishes of what you want crawled or not. Most bot's don't care or listen to it
But I understand bots like msn/google/yahoo mind it to a degree.
Logged
cathyslife
«
Reply #2 on:
September 07, 2013, 06:34 »
0
.
Logged
Print
Home
| Pages: [
1
]
« previous
next »
MicrostockGroup
»
Microstock Photography Forum - General
»
Symbiostock
»
Symbiostock - SEO & Marketing
»
What is robots.txt for?
Related Topics
Subject / Started by
Replies
Last post
Robots.txt and url server errors
Started by
cathyslife
Symbiostock - Technical Support
5 Replies
2133 Views
September 03, 2013, 15:46
by
cathyslife
Sponsors
Microstock Poll Results
Site
Earnings Rating
Top Tier - Big 4
Stocksy
AdobeStock
Self-Hosted
Shutterstock
Envato
Canva
Alamy
iStock
Indivstock
Pond5
DepositPhotos
Zoonar
Dreamstime
Crestock
EyeEm
Vecteezy
123RF
PantherMedia
Bigstockphoto
Canstockphoto
PhotoCase
500px
ClipDealer
VectorStock
YayImages
More Poll Results
Updated: 2024-06-16
Sponsors