Welcome,
Guest
. Please
login
or
register
.
Did you miss your
activation email
?
1 Hour
1 Day
1 Week
1 Month
Forever
Login with username, password and session length
Home
Forum
Help
Tools
Keyword Research Tool
Keyword Comparison
Polls
History Graph
Submit Vote
Support
Advertise
Premium Membership Sign-Up
Search
Login
Register
MicrostockGroup
»
Microstock Photography Forum - General
»
Symbiostock
»
Symbiostock - SEO & Marketing
»
What is robots.txt for?
MicrostockGroup Sponsors
« previous
next »
Print
Home
| Pages: [
1
]
Author
Topic: What is robots.txt for? (Read 3435 times)
0 Members and 1 Guest are viewing this topic.
aliceria
«
on:
September 07, 2013, 02:31 »
0
Search engine robots (bots) look at this file to see if they should crawl and index pages on your site, certain file types or even the entire site. An absence of this file gives them the green light to crawl and index your site
Logged
Leo Blanchette
«
Reply #1 on:
September 07, 2013, 02:41 »
0
Traditionally its there to give your wishes of what you want crawled or not. Most bot's don't care or listen to it
But I understand bots like msn/google/yahoo mind it to a degree.
Logged
cathyslife
«
Reply #2 on:
September 07, 2013, 06:34 »
0
.
Logged
Print
Home
| Pages: [
1
]
« previous
next »
MicrostockGroup
»
Microstock Photography Forum - General
»
Symbiostock
»
Symbiostock - SEO & Marketing
»
What is robots.txt for?
Related Topics
Subject / Started by
Replies
Last post
Robots.txt and url server errors
Started by
cathyslife
Symbiostock - Technical Support
5 Replies
2120 Views
September 03, 2013, 15:46
by
cathyslife
Sponsors
Microstock Poll Results
Site
Earnings Rating
Top Tier - Big 4
Stocksy
AdobeStock
GettyImages
Shutterstock
Alamy
iStock
Envato
Canva
Indivstock
Pond5
DepositPhotos
Dreamstime
EyeEm
MotionElements
Zoonar
ColourBox
Vecteezy
123RF
PantherMedia
SignElements
Pixta
Bigstockphoto
Canstockphoto
PhotoCase
ImageVortex
ClipDealer
FeaturePics
VectorStock
500px
ScanStockPhoto
Crestock
Clipcanvas
Dissolve
YayImages
MostPhotos
Self-Hosted
More Poll Results
Updated: 2024-05-21
Sponsors