Welcome,
Guest
. Please
login
or
register
.
Did you miss your
activation email
?
1 Hour
1 Day
1 Week
1 Month
Forever
Login with username, password and session length
Home
Forum
Help
Tools
Keyword Research Tool
Keyword Comparison
Polls
History Graph
Submit Vote
Support
Advertise
Premium Membership Sign-Up
Search
Login
Register
MicrostockGroup
»
Microstock Photography Forum - General
»
Symbiostock
»
Symbiostock - SEO & Marketing
»
What is robots.txt for?
MicrostockGroup Sponsors
« previous
next »
Print
Home
| Pages: [
1
]
Author
Topic: What is robots.txt for? (Read 3382 times)
0 Members and 1 Guest are viewing this topic.
aliceria
«
on:
September 07, 2013, 02:31 »
0
Search engine robots (bots) look at this file to see if they should crawl and index pages on your site, certain file types or even the entire site. An absence of this file gives them the green light to crawl and index your site
Logged
Leo Blanchette
«
Reply #1 on:
September 07, 2013, 02:41 »
0
Traditionally its there to give your wishes of what you want crawled or not. Most bot's don't care or listen to it
But I understand bots like msn/google/yahoo mind it to a degree.
Logged
cathyslife
«
Reply #2 on:
September 07, 2013, 06:34 »
0
.
Logged
Print
Home
| Pages: [
1
]
« previous
next »
MicrostockGroup
»
Microstock Photography Forum - General
»
Symbiostock
»
Symbiostock - SEO & Marketing
»
What is robots.txt for?
Related Topics
Subject / Started by
Replies
Last post
Robots.txt and url server errors
Started by
cathyslife
Symbiostock - Technical Support
5 Replies
2099 Views
September 03, 2013, 15:46
by
cathyslife
Sponsors
Microstock Poll Results
Site
Earnings Rating
Top Tier - Big 4
Stocksy
AdobeStock
GettyImages
Shutterstock
iStock
Envato
Canva
Alamy
MotionElements
Pond5
Vecteezy
Indivstock
DepositPhotos
Dreamstime
Zoonar
123RF
SignElements
ColourBox
EyeEm
PantherMedia
Pixta
PhotoCase
Canstockphoto
MostPhotos
Bigstockphoto
YayImages
ImageVortex
FeaturePics
Self-Hosted
Clipcanvas
ClipDealer
500px
Dissolve
ScanStockPhoto
VectorStock
Crestock
More Poll Results
Updated: 2024-04-24
Sponsors