Author Topic: Using Amazon S3 for storage and a Proof of Concept website (Read 13384 times)

Adeptris · « **on:** May 09, 2009, 02:00 »

I am looking to use Amazon S3 http://aws.amazon.com/ for a Proof of Concept website and I am very impressed with this service, so I thought I would share my experience, please note there is no affiliate link so this is not pimping for clicks

Amazon S3 is an on line file storage where you pay as you go, I created my account, note you have to register a credit / debit card so they can charge monthly, then you go to the website and retrive your keys, these are Your 'Access Key ID' and Your 'Secret Access Key' used to logon to your file folder.

Next step was to install an add on for Firefox, there are other ones for other browsers as well as Desktop solutions, I installed 'S3 Firefox Organizer' and selected the service from the tools menu, I had to enter a unique name for my 'Bucket' which is just a root folder.

My test of this service was to upload 129 images in 5 folders that I have prepared for Alamy, so these are 5100 x 3400, the 129 files total 775mb, average file size is 6mb, I selected folder sync. so as I add images I can sync. them to the Bucket, I then uploaded, I use Virgin (NTL) and the files went up at about 40 files an hour

Lets keep it simple and work in a unit of 100 = 600mb, and look at what it will cost me, storage is $0.15 a month per GB, and uploading and downloading is $0.17 per GB

The upload these 100 files was $0.17 / 0.600 = $0.3, to store these 100 files will be $0.25 a month ($3.00 a year per 100 images, $30 a year per 1000 images)

What I have for my money is full size images securely stored on-line and I can get access to these files on my laptop without having the need for large storage, I can have an off-site remote backup of all my files, I can also store other files like model releases, I can also now have a website that uses the images in the bucket to minimise the ISP bandwidth and storage.

The proof of concept website is to allow a group of photographer to keep ownership and control of thier own images via Amazon S3 and dispaly them in a virtual Agency environment, that will take a while to pull together but looks like a realistic option

The proof of concept website which I created yesterday so has no content yet, but is here if you want to bookmark it or get involved www.photographers-collective.com

This website will be DotNetNuke and will use Xmod to extend the modules and create the Custom Areas like pricing, it will use an Install of MyMediaGallery for the image rendering and the core Store product tables and functionsfor checkout and also will be used for the lightboxes, as this is proof of concept the Production website could be then built in any language php, asp etc:

David

Adeptris · « **Reply #1 on:** May 10, 2009, 03:05 »

After several discussions on the forum the concept is going through a transition and the new thinking is more along the lines of a service provider

As the Photographers put many images on their own sites and already have them with stock imaging sites we would not want to cut this revenue, we already have a mix of websites and services on different platforms this would make it very hard to have a standard website template, so we would need to look differently as a service, and at storage and retrieval services like Amazon Web Services, where each photographer would manage their images in a uniform structure via an applet which will allow uploading to Amazon as well as other services to extract the data and thumbnails.

The idea behind the collective is that we do not want to undercut each other, or drive down prices, but we do want to get a fair return, so we would still need to set as a collective the minimum prices these would be based on an existing service "Sales Price" where the image licence is already on sale, these prices and competition rules would need to be agreed by anyone using this model or service

Once we have formulated a template and structure, creating a desktop applet to upload to Amazon will not be hard as I have already done some windows image applications extracting and saving data and creating thumbnails, once we have data showing a large number of contributors and image counts available then Agents, Brokers, Buyers and website designers would be interested

The buyer or broker would be able to search for images and return the inventory data in very small packets with the thumbnails that matched the search in the agreed format to their desktop applet, website or catalogue, this would suit direct image buyers as they would get images in different price bands, so they can fulfil a Request.

As we are setting the 100% cost we want, it would be up to the agents and broker to set their commission level and not the other way around as it is now, so the dog would wag it's own tail.

So what we are thinking of is a system where the Photographer stores their own data via the desktop application in a set template format, this would be full size, comp, thumbnail and xml data packet, and we then give a set of free tools that would allow anyone to setup a website or service to use this data, and they would then instantly have data and images available as thumbnails and a system where the Photographer gets paid as the buyer, agent or broker calls off the full size image and licence.

A service like this would mean that Photographers can still use existing website services to sell licences for the same or different images, and also have other direct sales transactions that cut-out the middle man.

We still need to maintain a central site where brokers and buyers can download applets which will meed to develop as things change, and this site would also have the photographers central data and read only bucket access codes for data and thumbnails to facilitate the services.

Questions still are how do we organize to agree a structure to create the services, but if you now have an image on a website that charges $10 for a large size then we could add that image to the pool at $8-$9 to the Photographer, it is not about saving the buyers money but facilitating a central service that returns images at all prices.

The agents, buyers and brokers then get uniform data, the collective becomes a repository so developers can create a thumbnail search engine for stock images.

The plus points to this model is that the photographers will just be backing up their images in a structure and to a template for online storage in a format that will allow sales to be generated directly.

A buyer could setup an Amazon account and just download our desktop application, enter the image details and fetch thumbnails from all different photographers accounts to their own buckets, and the cost of this service is spread between Photographers and image buyers, the Photographer pays for their own storage bucket, the transfer of data between buckets is free, the broker or buyer covers the cost of getting the data from their bucket to their website or desktop, the sales could even be brokered by Amazon.

Cost to the Photographer to use Amazon S3 as a backup and collective service is about $35 a year per 1000 images, any sales could negate this.

David

alias · « **Reply #2 on:** May 10, 2009, 03:19 »

I like the direction that you are taking this. Great for thinking it this far through already.

But I am not convinced that additional desktop applications are the way to go. It needs to be a good web interface.

Also bear in mind that a great majority of graphic professionals are Mac users !

crazychristina · « **Reply #3 on:** May 10, 2009, 05:50 »

The implementation that you've thought through sounds very interesting. My concern is the 'we' bit. 'We' is effectively an agency, a decision making body that sets some standards, determines membership?, sets prices? etc. The politics and structure of the collective are probably the central issue here.

Peter · « **Reply #4 on:** May 10, 2009, 08:57 »

why pay when you have PhotoBucket and simmilar... for free?

Graffoto · « **Reply #5 on:** May 10, 2009, 09:24 »

Quote from: Peter on May 10, 2009, 08:57

why pay when you have PhotoBucket and similar... for free?

Wrong Peter, Photobucket is only free for lower resolution files. Note that the OP said that his files are around 6mb in size.

Photobucket charges approximately $40 per year to store files that are 5 mb in size.

Adeptris · « **Reply #6 on:** May 10, 2009, 12:46 »

Quote from: Peter on May 10, 2009, 08:57

why pay when you have PhotoBucket and simmilar... for free?

Peter, nothing is free someone is paying, the free space is paid for by the users that want the enhanced packages, when you join these services you soon see the limitations, with $33 and the Amazon bucket I can upload and store 1000 images at 5100 x 3400 for a year, now I have a flickr pro which is about the same and I only have a couple of hundred 500px longest side low res images uploaded, a lot of the bucket services do not store and host the images, they use services like Amazon and sell you a folder in thier bucket.

The way I would see this going would be that you can own your own bucket with amazon and pay only for your own use and not others, the object is to give the Photographers control of thier own assets, how many full resolution images are out there with the stocksites where you have just handed over most of your portfolio this is really about DAM (Digital Asset Management), and we just give ours out for others to manage.

David

Adeptris · « **Reply #7 on:** May 10, 2009, 12:56 »

Quote from: alias on May 10, 2009, 03:19

I like the direction that you are taking this. Great for thinking it this far through already.

But I am not convinced that additional desktop applications are the way to go. It needs to be a good web interface.

Also bear in mind that a great majority of graphic professionals are Mac users !

There are Plug-Ins for all Browsers that allow yo to upload to 'Amazon S3' so that is a route we could take, I used a Firefox plug-in for my test, we would not want to host a web page because of the bandwith Fee's, but we would need a bit more than a S3 Explorer, as you need to set other attributes like licences and releases, thanks it is something to think more about and I am bouncing idea's, as the Photographer would be using thier own bandwith and paying for thier own storage bucket on a "Pay as You Go" basis there are no costs at this point only the development of the Plug-ins, and this option would suit all operating systems, and I don't think a buyer would mind using a plug in to manage thier requests.

David

Adeptris · « **Reply #8 on:** May 10, 2009, 13:17 »

Quote from: averil on May 10, 2009, 05:50

The implementation that you've thought through sounds very interesting. My concern is the 'we' bit. 'We' is effectively an agency, a decision making body that sets some standards, determines membership?, sets prices? etc. The politics and structure of the collective are probably the central issue here.

Averil, I am using the we and not I because if this does become a production model then it would require more than just me, I am talking just of a service and set of tools, that links Suppliers to Customers, not an Agency or Merchant but a generic template to work to, and functions via dll objects that can be built into website templates and applications.

The stumbling block at the moment is how to make the development and a search engine service pay it's own way, and allow the sales side of the service to expand free of charge for organic or viral growth.

Think of it more in terms of a Search Engine for submitted images, now Google scans pages and creates thumbnails and search strings, we cannot do that with images on servers, so we need a submission to add an image to the search engine, and a way of presenting thumbnails and data to potential clients, so this is where a template would be used, there needs to be clear rules for using a service just like the T&C when you submit to a new stocksite, there could be no Inspection or QC but a flag which would say if an image had beed accepted to a stocksite without naming which site, and a method of reporting abuse resulting in the deletion of search entries for a contributor.

Taking Google as a model where the search is free but the advertisers pay for the submission that is likely the best way to monertize the search engine, so you would create your own Amazon S3 bucket, install an image management plug-in in your browser enter your key, manage your images and submit these to the database, you would be charges a small monthly fee for your entries to the database, this pays for the development and movement of data etc, as this will be on a browser plug-in basis, there are no bandwith fee's but there are fees to retrieve the data which is part of the subscription.

David

le_cyclope · « **Reply #9 on:** May 10, 2009, 13:35 »

This whole concept seems very interresting. And for me, since I do not have any talent in web design (but have lots of other talents

) I would even agree to pay for a standard web template which would be usable without any (or little) modification.

Selling this template could be a way to generate cash flow.

Claude

alias · « **Reply #10 on:** May 10, 2009, 13:39 »

1. Have you got any ideas about how payment would work?

2. I do not think you should require buyers to use a browser plugin. I normally don't use sites which require me to install plugins and I know that many people feel the same way. So it becomes almost like a marketing issue. Mac users do not even have to install Flash on Safari (it's there and updated via Apple updates). Many graphics houses and offices do not allow users to install software.

^ but I have a lot of respect for what you are trying to do and the way in which you are thinking this through

Adeptris · « **Reply #11 on:** May 10, 2009, 13:46 »

Just an addition, like a steam train this one once you get thinking:
I do think that if we can produce a set of generic dll (Dynamic Link Libraries) tools that many can use on thier own websites or browsers, we may make the big sites look again at thier own models, also there is potential to open up new markets, look at the number of online news, debates, specialist blogs out there, lets say you are a blogger and you are writing a recipe on a blog for a dish of the week and you need an image, you have the plug-in installed, put the dish name in "Teriyaki Chicken", maybe select a price band, search comes back with a list you can narrow the search view thumbnails and purchase the image from your browser, the funds are in the Photographer's paypal or Amazon account and the image, Licence and T&C are on the buyers browser plugin to download, a copy of the images would be moved from your bucket to a generic buyers bucket for 5 days, then deleted.

Look at this from an overheads perspective we have not gone to a website, and used the buyers bandwith, so the photographer gets the cream!

David

Adeptris · « **Reply #12 on:** May 10, 2009, 14:01 »

Quote from: alias on May 10, 2009, 13:39

1. Have you got any ideas about how payment would work?

2. I do not think you should require buyers to use a browser plugin. I normally don't use sites which require me to install plugins and I know that many people feel the same way. So it becomes almost like a marketing issue. Mac users do not even have to install Flash on Safari (it's there and updated via Apple updates). Many graphics houses and offices do not allow users to install software.

^ but I have a lot of respect for what you are trying to do and the way in which you are thinking this through

There are two fields of thought here with the Plug-in vs browser, both have a place, lets look at the plug-in first, I posted just after you, I have brought Images from several sites to use on my own websites, I would have no problem installing a plug-in that allowed me to select and purchase the images in one place, I did not care which stocksite I was only interested in getting the images I wanted, there is a potential new market, one which returns the best percentage to the photographer.

If this is designed well then new merchants will take up this model use the SDK (software developers kit), select and mark images for inclusion in thier libraray, here again is a new market for photographers, no more uploading many times changing information for different sites, just upload once complete the template, based on your template and plug-in selections if the image matches any merchant's template then they will be sent a link, they can then view the comp and one click include the image in thier library which you can accept or decline, as thier website will be designed to use the toolkit, you would not have to do anything and the price you set would need to be paid in full before any image is transfered.

David

Adeptris · « **Reply #13 on:** May 10, 2009, 14:05 »

Quote from: le_cyclope on May 10, 2009, 13:35

This whole concept seems very interresting. And for me, since I do not have any talent in web design (but have lots of other talents ) I would even agree to pay for a standard web template which would be usable without any (or little) modification.

Selling this template could be a way to generate cash flow.

Claude

That would be another good way of generating revenue to cover development costs and would give you a standalone website that looked at the templated data in your own bucket.

David

alias · « **Reply #14 on:** May 10, 2009, 14:29 »

Quote from: Adeptris on May 10, 2009, 14:01

There are two fields of thought here with the Plug-in vs browser, both have a place, lets look at the plug-in first, I posted just after you, I have brought Images from several sites to use on my own websites, I would have no problem installing a plug-in ...

If you build a system that requires a plugin then only people who are okay with plugins will use it. If you build a system that does not require a plugin then you have a bigger potential customer base. Most graphics houses and media pros are using Macs. Many graphics houses and companies will not allow people to install plugins.

Quote from: Adeptris on May 10, 2009, 13:46

I do think that if we can produce a set of generic dll (Dynamic Link Libraries) tools that many can use on thier own websites or browsers

Are you proposing building a system which would have to be deployed on servers running MS Windows ? If so then I think that would be a mistake. I think that the solution needs to be platform neutral.

Adeptris · « **Reply #15 on:** May 10, 2009, 15:39 »

Quote from: alias on May 10, 2009, 14:29

[Are you prosing building a system which would have to be deployed on servers running MS Windows ? If so then I think that would be a mistake. I think that the solution needs to be platform neutral.

Hi alias,
I am just brainstorming for now, put all the ideas in and sort them out, there have been some good points and the model is changing by the hour, looking at the positives and negatives of each point then try to come up with a workable solution, which is platform independant, I have been looking at what Amazon have, this is "Pay as you Go and only for what you use", the main problem I see is changing the photographers perspective that everything is free,and getting all to pay a few $$ to store thier images and template them in a single structure that any new sites can use, each site now has it's own quirks, payment structure and rules, this means that a photographer may have to upload several times, the system I would like to see is my image as a product in it's own right and there being no need to create at my own cost in time different variants for different merchants.

The merchant or buyer can then become "My Agent" and sell a licence on my behalf, this licence will have the same purchase costs to any agency, the competition is then how good are the agents at keeping thier overheads down, not how hard can they squeeze the photographers.

Look at what happens now the Photographer thinks "It is free to upload to stocksites, I will upload everything", this attitude and concept is the reason the retuns are so poor and photographers with and without sales are complaining, it is a business and nothing is free, but now with the massive growth and a few Photographers are paying for the many plus the overheads to manage others images that may never sell, also our digital assets are just spread all over the place in folders on servers all over the world.

The model I am looking at is in use now with some pay services, the difference is that they sell you space in thier bucket, the price they charge is inflated to allow them to offer free space to others and hope to get them to upgrade later, once again the few paying Photographers fees are paying for the service usage for the "free account holders", I would change this so each photographer pays for thier own usage as they go, but gets a much better return with a sale.

Below is a programmers link that might give an overview of what Amazon could offer, it takes a while to get behind it but I will have a go at a simple overview, you use thier services to do the following without the need for any IT department, you run thier virtual server that your services talk to, this server has all the software running and is polling the queue for messages, lets say you upload an image and its attributes, the image would get resized as a thumbnail and a composite size with the original and loaded into your bucket, a data packet would be sent which would create an entry in the search database.

When a buyer enters a search string on a webpage this string is sent as a packet, the queue recieves the packet and runs a search returning any matching entries.

David

http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1602&categoryID=152

Adeptris · « **Reply #16 on:** May 11, 2009, 14:12 »

I started thinking along the same lines as hqImages a collective website, but I have repositioned my thinking and will be going along the lines of an indexed search engine that returns thumbnails from a search string through a software development kit, I have done a lot of research this weekend and have a rough idea how to do this using Amazon Web Services, if the web engine tools are freely distributed then they could be easy built into websites and templates.

This option does require uploading to Amazon but only once and it is pay as you go for useage, when you upload and complete the data for the image this would include released, licence type and cost which is your share and also set a maximum sales price etc: then you can submit it to the search engine, all that the search engine does is accept a packet of data and produce a thumbnail and composite image in your folder, no images are transfered because the data packet knows where the images are in your bucket.

Now if you submit your images as private then you can use the sellers toolkit to facilitate searches on your website, if you make your images public then other services can host your thumbnails, that is agencies and merchants sites you have agreed to, and you would be able to offer a further discount at this stage, but you can set the price you want, or you could use a built in global "Fair Price", this flag would be searchable and give agents a budget value to work with.

Now any service has to have revenue and this would be funded by micro payments by the photographers and sellers, buyers pay nothing, not sure how much but I would think $0.01 for each thumbnail of yours retrieved by a search, and you can set a maximum spend per day or month, for a sale the micro payment would be between $0.025 and $0.05 depending on transfer costs.

Now how would you get paid, there are two ways, firstly for private sites you can handle the transaction collect payment and send the Image, or for both Private and Public images the transaction could be handled by the toolkit the search engine would not be holding your money at any stage, as soon as a payment recieved request comes in the full size file would be transfered to the buyer, and full payment will already be with the photographer, no more waiting for a payout band, this action would be triggered by a IPA payment reply from PayPal or Amazon both of these have facilities for Micro Payments, also remember if the sale through an agent was $5 you would likely be getting $4 instantly.

What we have with this model is organic growth and fair trade, a system where you only pay for what you use and that does not require you to upload your assets to many websites like you do now, upload once and your image could be on many specialist sites within hours, there will be people that try to cheat the system by changing the uploaded files, but as the image is uploaded and submitted to the database a checksum will be calculated for the image, comp and thumbnail, if this has changed when a sale takes place the transaction will be declined, photographers would register with the service with a link to thier images, any porn would not be accepted and if reported the account suspended.

David

kgtoh · « **Reply #17 on:** May 12, 2009, 14:51 »

I think this is a fantastic idea, both from a technical and business perspective.

While concepts are still at an early stage, I think this is has enormous potential and could really revolutionize distribution of intellectual products.

melastmohican · « **Reply #18 on:** May 12, 2009, 16:14 »

You start looking in direction I like more than initial idea.

Take a look at this example: iObjects - Monetize your Amazon S3 objects using Amazon FPS

Adeptris · « **Reply #19 on:** May 13, 2009, 02:54 »

Quote from: melastmohican on May 12, 2009, 16:14

You start looking in direction I like more than initial idea.

Take a look at this example: iObjects - Monetize your Amazon S3 objects using Amazon FPS

Hi,
I have looked at this and a number of other examples, this one requires Amazon "Flexible Payments Service" this is only available to US Credit Card holders not EU, UK or ROW, this whole concept will succeed or fall on the Amazon tools and creating a database search engine, so the first proof of concept I am looking at is using 11000 words to create about 5-7 million clustered search entries of random keyword combinations of between 7 - 10 words, I should be able to do this within the next two weeks, I have skills in vb.net and other languages but I will need to look at C#, Java, Ruby, Rails and SOAP.

Then by create a web page with a simple search box to test how fast the search will be, and allow this to be tested by anyone on my website, if I can get this up and running and the backend software lives up to expectation, then I think some people that might be sitting on the fence, may climb down and get involved in creating a shared project where we all can be the owners, it is not just programming skills that would be needed, the project would be broken down in segments which would require project leaders to get the tasks completed.

David

melastmohican · « **Reply #20 on:** May 13, 2009, 10:41 »

I did not check if Amazon offers any sort of database tools. As far as programming languages are concerned I would rather go with C# or Java, these are most popular so it would be easier attract developers. I would recommend to use Google Code for hosting source code.

Adeptris · « **Reply #21 on:** May 13, 2009, 14:56 »

Quote from: melastmohican on May 13, 2009, 10:41

I did not check if Amazon offers any sort of database tools. As far as programming languages are concerned I would rather go with C# or Java, these are most popular so it would be easier attract developers. I would recommend to use Google Code for hosting source code.

Amazon have all the tools required and the infustructure to deliver, the services you pay only for usage and they will grow with the solution, I prefer Amazon as Google is more about thier own API's that tap into thier services.

Google crawls website and then Indexes images but you cannot include seraches for all the other data you need for commercial images, google are more about social web users, if they were more commercial they would index the images from all the agencies, do a google search select images and put in some keywords, in my tests the only commercial images come from FotoSearch and Inmagine they have likely paid for inclusion.

David

MicrostockGroup Sponsors

Author Topic: Using Amazon S3 for storage and a Proof of Concept website (Read 13384 times)

Adeptris

Adeptris

alias

crazychristina

Peter

Graffoto

Adeptris

Adeptris

Adeptris

le_cyclope

alias

Adeptris

Adeptris

Adeptris

alias

Adeptris

Adeptris

kgtoh

melastmohican

Adeptris

melastmohican

Adeptris

Related Topics

Sponsors

Microstock Poll Results

Sponsors