MicrostockGroup

Microstock Photography Forum - General => Symbiostock => Symbiostock - Technical Support => Topic started by: ShazamImages on August 28, 2013, 12:52

Title: Tips on How to Administer a Production Symbiostock Website
Post by: ShazamImages on August 28, 2013, 12:52
I have been in the IT industry for a few decades and thought that I would share some tips on how to administer a Production website properly.  Hopefully some of you will find this useful:

Many Symbiostock artists have never had a website before and therefore don't understand any of the standards associated with the IT industry.

Your Symbiostock site is an e-commerce site.  It's sole purpose (for the majority of artists), is to sell your artwork.

Your Symbiostock site should also be what is referred to as a "Production" website.  You should never be developing or testing on your Production website.  You should setup another website for development and testing.

Your Symbiostock website should be up and running 99.999% of the time.  This is commonly referred to as the "five nines": http://en.wikipedia.org/wiki/Five_nines (http://en.wikipedia.org/wiki/Five_nines)

FYI: If you are up for 99.999% of the time, then you will only be down for a little over 5 minutes per year! 

If your website is down when a customer comes to it for the first time, they might never return again.  If a customer is returning to your website while it is down, they might never come back again.  Either way, that could end up in lost sales.  This is not something that you want to happen.

Here are some "tips":

Backups:
   - Backup your website often.  If your website crashes, you can't recover unless you have a backup.  How often you backup your website is a decision that you need to make, but once a day is a good starting point.
   - Backup your website during "off-hours".  "Off-hours" is a term used to denote a time when there is very little traffic on your website.  You don't want to be running a backup while customers are using it, because it might slow down your website (since it is using resources to do the backups) and the customer might get annoyed and leave.  You also want to do a backup when traffic is slowest so that the data in your backups will be "consistent" (see here: http://en.wikipedia.org/wiki/Data_consistency (http://en.wikipedia.org/wiki/Data_consistency)).   Evenings and weekends are usually the slowest time for most websites, but you need to figure out what is the slowest time for your website and then schedule backups during that timeframe.  Remember that a website can be accessed from all around the world, not just the country where you reside.
   - Backup your website before you make ANY changes.  The last thing you want is to make a change and then receive the "white screen of death".  If your website goes down and it takes you hours (or days) to bring it back up, that will most likely lead to lost sales and annoyed customers.
   - Keep as many backups as you can for as long as you can.  You never know when you will need to recover from a backup that was from a week or two ago.  Sometimes updates look like they go well at first, but then you find a serious issue a week or more later.  Remember, you can't recover if you don't have the appropriate backup.

Maintenance (also commonly referred to as "Change Control")
   - Maintenance includes any sort of change to your website, including adding, modifying, and deleting plugins, widgets, themes, etc.
   - All maintenance should be fully tested on a non-Production website.  You don't want to use your Production website for testing purposes.
   - Maintenance should only be delivered during off-hours.  Once again, you don't want to make a change to your website and then have it go down for hours during your time of highest traffic.
   - All maintenance deliveries should have a "rollback plan".  A "rollback plan" is a way to reverse the changes that you made, just in case things don't go as planned.
   - Try to withhold doing updates right after they come out.  This is called being on the "bleeding edge" (and there is a reason for the terminology).  Wait at least a few days or even a week or two before applying an update.  This will give the developers a chance to find any bugs that they couldn't find in their testing phase.

Please note that this isn't an all-inclusive list, but rather just a starting point.

Hopefully others that have been in the industry will chime in and add some "tips" as well.
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: Kerioak~Christine on August 28, 2013, 14:26
My host does regular backups but after the Justhost/bluehost/hostmoster fiasco the other week I decided to back up the site to my computer.  It took me two days of attempts and the final - and  successful backup took nearly six hours to download.  Not something I want to try every day.

With most hosts I think you can set up sub-domains to do your testing on - I have main and testing site on subdomains
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: ShazamImages on August 28, 2013, 14:33
...I decided to back up the site to my computer.  It took me two days of attempts and the final - and  successful backup took nearly six hours to download.  Not something I want to try every day.

The first time I did a full backup, I had the same issue.  It took hours to download.  In addition, when I tried to schedule it with GoDaddy, GoDaddy killed it after about a minute (I guess they feel that anything that is taking that long is a runaway process).

So I then decided to break up the backups into "segments".  This has allowed me to download the backups to my local computer in about 10-20 minutes each day (even though I have uploaded a lot more images, added plugins, etc).
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: stockphoto-images.com on August 28, 2013, 15:25
Wow, thanks ShazamImages for your thread about this.

I have a very important question I want to ask:

I'm with Hostgator and they offer free backups as long as your shared account is below 20GB capacity and I'm just below that at the moment.

However, if I have to make a backup myself, do I have to ZIP and download my entire Wordpress installation - 20GB of stuff?

And then re-upload that in case of a failure?
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: ShazamImages on August 28, 2013, 16:18
Wow, thanks ShazamImages for your thread about this.

I'm glad that you found the information interesting.

However, if I have to make a backup myself, do I have to ZIP and download my entire Wordpress installation - 20GB of stuff?

When you zip up the files, it will compress them somewhat.  The amount of compression depends on the type of file (and the type of compression used).  Text files (such as source code for the Symbiostock theme, plugins, etc) will usually compress around 90-95%.  Images will usually only compress about 5-10%, since they are already compressed file formats.  So your backup will be less than 20 GB, but how much less is unknown.

Anyway, yes you would need to download the entire amount the first time.  You could break up the backup into smaller chunks, and then backup each chunk separately.  That might make it more manageable.

But subsequent backups could be much smaller if you break up your backups into "segments".

For example, you don't need to download ALL of your images every day.  You just need to download the images once (or maybe twice for extra security).  After that you only need to download any new images that you upload.  Symbiostock renames your images and appends a number to the beginning of the image file name.  So you could backup all of your images where the file name begins with 100 to 199 in one backup, and then backup images where the file name begins with 200 to 299 in another backup, etc.  Once you back those images up and download them, you don't need to back them up again.  So the backups become much more manageable (and take less time downloading).

I have broken up my backups into five "segments":
- Original Images
- Generated Images (thumbnails and watermarks)
- Other Images
- Everything Else (source code, plugins, etc)
- MySQL Database

And then re-upload that in case of a failure?

Yes, unfortunately you would have to upload the entire thing in order to do a recovery.  The good news is that you would be able to do a recovery.  If you didn't have the backups, you might not be able to recover and you would have to start from scratch all over again!
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: CrackerClips on August 28, 2013, 17:27
Good information but I'm a little confused about backups.  I've always done a database backup using the plugin WordPress Database Backup but I honestly don't know how to backup my entire site.  My host (godaddy) has always told me that they do a backup of the entire site every day but I know that may not be entirely reliable.  So how do I go about backing up the entire site?
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: Anita Potter on August 28, 2013, 17:30
I don't think I go about it the right way but I log in to my ftp and throw it into an update folder.  Then I go and backup my database.

Is the general rule:  back up every time you add and/or make changes? or Daily backup (not sure how I could do that with just updated files instead of the entire site if that's even possible) or update once a week?
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: Ron on August 28, 2013, 17:31
Can I just drag the www folder onto my 1TB ext HD? Will that be a proper back up?
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: ShazamImages on August 28, 2013, 17:40
Is the general rule:  back up every time you add and/or make changes? or Daily backup (not sure how I could do that with just updated files instead of the entire site if that's even possible) or update once a week?

It's both.  You should backup on a regular basis (e.g., every day) AND backup before you make any changes.  Of course, you could always schedule your changes after your regular backups are done and thus satisfy both "rules".

Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: ShazamImages on August 28, 2013, 17:44
Can I just drag the www folder onto my 1TB ext HD? Will that be a proper back up?

If your www folder also contains your MySQL database backups, then you should be able to do that.

Otherwise you would need to also copy the database backups.

While that solution is easy, the problem is that it can take a LONG time if you have to do that on a daily basis and you have a lot of images.
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: ShazamImages on August 28, 2013, 17:49
Good information but I'm a little confused about backups.  I've always done a database backup using the plugin WordPress Database Backup but I honestly don't know how to backup my entire site.  My host (godaddy) has always told me that they do a backup of the entire site every day but I know that may not be entirely reliable.  So how do I go about backing up the entire site?

You would need to create a script (on the host) that would backup all of the files into one (or more) file(s) and then download that file (or files).  And the backup would need to consist of the MySQL database and all of the web files (WordPress source code, Symbiostock source code, plugins, themes, images, generated images, etc).
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: CrackerClips on August 28, 2013, 18:02
What about using a service like Back Up Machine (http://www.backupmachine.com (http://www.backupmachine.com))?  Are there better services?
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: ShazamImages on August 28, 2013, 18:05
What about using a service like Back Up Machine ([url]http://www.backupmachine.com[/url] ([url]http://www.backupmachine.com[/url]))?  Are there better services?


I'm not familiar with it, but it looks promising.  Maybe someone else can chime in.
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: stockphoto-images.com on August 28, 2013, 19:59
Thanks for your answer ShazamImages! It's much appreciated.

You are correct, in case of a loss, even a 22 GB backup will be less work than starting from scratch.

Have you personally automated any backup routines? Or do you do it manually every night for example?

I did backup my Wordpress site before, sporadically every few months and with way less disk space (couple hundred Mega Bytes).

Let's assume I'm not currently adding any new content, all I have to do is backing up my SQL database on a daily basis and nothing else?
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: quailrunphoto on August 28, 2013, 20:28
Thanks for your answer ShazamImages! It's much appreciated.

You are correct, in case of a loss, even a 22 GB backup will be less work than starting from scratch.

Have you personally automated any backup routines? Or do you do it manually every night for example?

I did backup my Wordpress site before, sporadically every few months and with way less disk space (couple hundred Mega Bytes).

Let's assume I'm not currently adding any new content, all I have to do is backing up my SQL database on a daily basis and nothing else?

This is a really great topic.  Something that I considered when I was looking at where to host my site.  BlueHost does automatic backups, daily, weekly and monthly.   But they only keep the latest of each.  Fortunately they have an easy zip download for these, so I just have to plan ahead and download the zip before I do an update.  I also have a test site just for playing with new updates, so I have a better idea of what may happen.

You might want to check with your host first to see what is available from them, it may even be included in your current services.

Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: Leo Blanchette on August 28, 2013, 20:39
What about using a service like Back Up Machine ([url]http://www.backupmachine.com[/url] ([url]http://www.backupmachine.com[/url]))?  Are there better services?


This person here offered Symbiostock backup services: http://www.microstockgroup.com/profile/?u=712 (http://www.microstockgroup.com/profile/?u=712)
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: ShazamImages on August 29, 2013, 05:35
Have you personally automated any backup routines? Or do you do it manually every night for example?

Yes, I have automated all of my backups.  I haven't automated the downloads yet though, so I download the files every morning or so.

Let's assume I'm not currently adding any new content, all I have to do is backing up my SQL database on a daily basis and nothing else?

Assuming you don't add any new images, don't add any new plugins / widgets, don't upgrade the theme, etc.  Then yes, all you would need to backup would be your MySQL database.

The only files that I have seen change on a regular basis are the CSV and XML files (for your website, your network's websites, and the websites in the network directory).  But I think that you can regenerate those rather easily.
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: ShazamImages on August 29, 2013, 05:40
Can I just drag the www folder onto my 1TB ext HD? Will that be a proper back up?

If your www folder also contains your MySQL database backups, then you should be able to do that.

Otherwise you would need to also copy the database backups.

While that solution is easy, the problem is that it can take a LONG time if you have to do that on a daily basis and you have a lot of images.

One more thing that I forgot to mention...

You can ignore the symbiostock_xml_cache directory from your backup (which will reduce the size of your backups by quite a bit).
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: Redneck on August 29, 2013, 08:51
Why don't you guys just use a cloud service for backup? I have my WP site set up to do a daily backup and to save it at a cloud. Automatically and in the background. I just check it weekly if all backups went well and have been stored/transmitted.
Other than that, it's set and forget and it doesn't cost money or any of your computer ressources.
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: Kerioak~Christine on August 29, 2013, 08:57
I have some  cloud storage but they are either small or will run out when I stop paying for subscriptions (they are linked to other things) so I ignore them.  I suppose I also don't really trust things I cannot hold in my hand in some way.    How do you get free cloud storage large enough for a symbiostock back up?
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: stockphoto-images.com on August 29, 2013, 09:16
Why don't you guys just use a cloud service for backup? I have my WP site set up to do a daily backup and to save it at a cloud. Automatically and in the background. I just check it weekly if all backups went well and have been stored/transmitted.
Other than that, it's set and forget and it doesn't cost money or any of your computer ressources.
Maybe it makes sense if you would create a separate thread that contains a tutorial how to do Symbiostock/Wordpress backups using cloud storage. Might be useful for a bunch of folks.

I'm totally new to cloud services as I do not trust them, but I guess since the NSA and half of the world already knows everything about me anyway, I might as well get cloud storage as well.  :P
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: Redneck on August 29, 2013, 09:21
Here is how you can get 20GB cloud storage for free: http://picturebreeze.com/stock/photos/20gb-free-cloud-storage/ (http://picturebreeze.com/stock/photos/20gb-free-cloud-storage/) . You can actually get a lot more than that when you refer people to this service later.

But I don't even use this one. I use the free Dropbox (2GB) out of convenience. The key is that I don't back up all images. Well, I don't back them up from my website is what I mean. What I do backup are the database tables and the plugins. My images are stored at a different hard drive and a different (paid) cloud service.
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: Anita Potter on August 29, 2013, 09:23
My host offers cloud as well but when they screwed up my migration and didn't migrate my database I don't really trust them with clouding my backups.
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: Redneck on August 29, 2013, 09:26
Why don't you guys just use a cloud service for backup? I have my WP site set up to do a daily backup and to save it at a cloud. Automatically and in the background. I just check it weekly if all backups went well and have been stored/transmitted.
Other than that, it's set and forget and it doesn't cost money or any of your computer ressources.
Maybe it makes sense if you would create a separate thread that contains a tutorial how to do Symbiostock/Wordpress backups using cloud storage. Might be useful for a bunch of folks.

I'm totally new to cloud services as I do not trust them, but I guess since the NSA and half of the world already knows everything about me anyway, I might as well get cloud storage as well.  :P

I might do that. Just need to find 20 minutes.
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: Redneck on August 29, 2013, 09:26
My host offers cloud as well but when they screwed up my migration and didn't migrate my database I don't really trust them with clouding my backups.

Saving the backup on your host site doesn't really make sense, does it?
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: Anita Potter on August 29, 2013, 09:32
Nope redneck it surely doesn't.  I don't even know how the cloud thing works either so I'll just spend around an hour backing up my own stuff to my computer.

Maybe a tutorial on how to back up a database and then how to restore it would be good as well.
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: Redneck on August 29, 2013, 09:36
I'll try to write an article about how I do (actually how I have them done because I don't do anything anymore) my backups automatically soon.
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: ShazamImages on August 29, 2013, 09:46
The key is that I don't back up all images. Well, I don't back them up from my website is what I mean. What I do backup are the database tables and the plugins. My images are stored at a different hard drive and a different (paid) cloud service.

If you don't backup the images from your website, how would you be able to do a proper restore if stuff hit the fan and your website was unrecoverable?  When you upload and process images, they get renamed by Symbiostock.  In addition, thumbnails and watermarked images are created.

If you don't have everything, then you would basically have to upload your images all over again from scratch.  Wouldn't you?
Title: Re: Tips on How to Administer a Production Symbiostock Website
Post by: Redneck on August 29, 2013, 10:14
You're right, the original images get renamed, too. I hadn't even realized that until today. I guess I need to rework my backup strategy and do a few test runs for the entire site including full size images. I'll let you know how it goes.