/First the good news. The work our team has done on the servers has really improved things. We"ve greatly increased the capacity of our systems both by increasing the number of servers that we are operating with, as well as reorganizing how the data on the database cluster is accessed and updated. We have also brought back many of the features that had to be disabled. We"re handling more transactions per second with far higher throughput than ever before. We served 2 billion thumbnails in September.
The challenge? Site traffic keeps growing at higher and higher levels. How much higher? An extra 1.5 million visits over the last 30 days, as compared to the 30 days prior to that. That works out to an extra 16 million page views. We"re serving over 2x what we did last year. Although we"ve made lots of progress, with such rapid growth there is more to be done.
We did foresee these increases, and took steps to accommodate them. In July and August we quadrupled our server capacity. However, some parts of the site cannot be scaled by adding more servers - they need to be centralized on a single server. This is the master server that has been talked about so much in the forums of late. We underestimated the effects of the increased traffic would have on the master - and did not anticipate the locking issues. The databases are still getting overloaded from time to time. Our internal experts and consultants are testing some operating system changes that we hope will help in the short term. They will continue working on it throughout the weekend.
We are also working on a long term plan to re-architect the system so we won"t rely on a single master server like we do now. We are looking at new technologies and new techniques. It will involve a lot of work, both in our code and in the way we store the data. But in the end, we will end up with a system that will be much larger in scale and scope, taking us to a whole new level.
We also have some new hardware en route to do some testing, but this will take time.
What this means
We may need to restart our main servers on occasion. This means no site for about 14 (agonizing) minutes if all goes well. Please understand that we realize this is unacceptable, and we"re doing everything in our power to keep it from happening. When it does, we ask you to be patient. If you phone into customer service, please be gentle. They are fielding hundreds of calls, and can"t tell you much more than our ETA to be back up.
To our customers
Again, we thank you for your patience. We will try to keep you updated, but please understand our time is best spent working on the issues. When things are ironed out, and we"re back to running normally, we can explain and expand further.
We appreciate your support. Though we know you are still experiencing painful moments, all of this is moving us toward an amazingly strong new iStock infrastructure. Please stick with us. You wonât be disappointed.
To our artists
We know you are worried about potential loss of royalties. Please do not be: we are selling far more images than we ever have before, even with the site issues.
For those worried about missed transactions. It is possible that a fraction of a percentage of orders are not properly processed when the site starts acting up. We"re investigating, and if we find it to be an issue, we will do a full audit to make it right. Rest assured you"ll be properly paid for images sold.
Again, we appreciate everyone"s understanding.