Nav: Home
Welcome, Guest. Please login or register.
Did you miss your activation email?
May 27, 2012, 04:04

Login with username, password and session length

MicrostockGroup

Microstockgroup Sponsors


« previous next »
Pages: [1] Print

Topic: FTP upload issues on most Stock Sites  

(Read 1473 times)
dnavarrojr


Dreamstime GaugeiStock Gauge
« on: May 06, 2009, 11:00 »

I am running into major upload issues with most of the stock video sites.  Because video files are so large, they can take a VERY long time to upload depending on your connection speed.

Using CuteFTP or FileZilla, I run into an issue that if left unattended, they will think that a file has not been successfully uploaded and will attempt to re-upload the file.  On a number of sites (such as Shutterstock) this results in many duplicate copies uploaded of the same file.

Since I am a programmer, I decided to write my own FTP program to try and diagnose the issue.  And what I discovered is that on those sites where I have the problem, the FTP server closes the control connection while the file is uploading.  So when the upload is done, the client software gets a time-out error trying to talk to the FTP server through the closed connection.  Most FTP clients assume this means the file was not successfully uploaded and attempt to upload the file again.

The root of the problem is actually the fault of the Stock Sites.  FTP servers are NEVER supposed to close a control connection if there is an active data connection.  The stock sites are not enabling (or disabling) this feature in their FTP servers.

I have tried to discuss this with the sites I have these issues with: Shutterstock, Fotolia, Clip Canvas, Can Stock and Dreamstime with little success.   Either they don't understand what I am talking about, or they don't care.

So my only solution is to try and find a client program that does not attempt to re-upload files. 

Does anyone have any suggestions?  This is not so much an issue with photos as they are smaller and can be uploaded before the control connection is closed.  Only when you start uploading 200 megabyte files and larger does this issue rear its ugly head.


Ignore | Logged


DepositPhotos.com
alias


« Reply #1 on: May 06, 2009, 11:16 »

What if you just open the terminal application and use ftp direct from the bash command line ?

Logged


donnelt
« Reply #2 on: May 06, 2009, 11:19 »


Have you tried the 'keep-alive' option in Filezilla->Edit->Settings->FTP?


Ignore | Logged


dnavarrojr


Dreamstime GaugeiStock Gauge
« Reply #3 on: May 06, 2009, 11:34 »

What if you just open the terminal application and use ftp direct from the bash command line ?

Same basic problem..  The control connection is closed by the server and the FTP utility closes with an error after uploading the first file.  The good news is that duplicates aren't uploaded, but you have to write a script (or batch file as the case may be) to upload batches to account for the error.


Ignore | Logged


dnavarrojr


Dreamstime GaugeiStock Gauge
« Reply #4 on: May 06, 2009, 11:38 »


Have you tried the 'keep-alive' option in Filezilla->Edit->Settings->FTP?

The problem with FileZilla (and CuteFTP and a half-dozen other programs I tried) is that it does FTP transfers multi-threaded.  So keep-alives work in the primary thread on some servers (not all) but in the data thread the control connection is still closed and it still causes FZ to re-upload the file.

I have tried to bring this to the attention of the FZ author but because the underlying issue is a bug or misconfiguration of the server software, he refuses to do anything about it.  He's a brilliant programmer, but an incredible ass with very poor social skills.


Ignore | Logged


donnelt
« Reply #5 on: May 06, 2009, 11:52 »


Set the time-out to zero and that may force the server to keep the control connection open.  (Edit->settings->transfers).

My control connections drop regularly during upload but I don't get a transfer failure at the end of the upload.  I assume you are using Passive?


Ignore | Logged


dnavarrojr


Dreamstime GaugeiStock Gauge
« Reply #6 on: May 06, 2009, 11:57 »


Set the time-out to zero and that may force the server to keep the control connection open.  (Edit->settings->transfers).

My control connections drop regularly during upload but I don't get a transfer failure at the end of the upload.  I assume you are using Passive?

Yup, passive mode.  I'll give that a try, but on many sites keep-alives don't work because the server closes any control connection that does not initiate a transfer within 3 to 5 minutes.  So, if I am uploading a file that takes 30 minutes to upload and FZ (or my own test program) is sending keep-alive commands the control connection is still closed because it doesn't initiate a new transfer.

For the moment I am just using my own program that I wrote and having it ignore the connection error.  That pretty much works, but I wrote it as a command line utility, so I have to stick a GUI on it to make it more useful.  In the meantime, I'm still hopeful of finding something else that works.


Ignore | Logged


stockastic


« Reply #7 on: May 06, 2009, 18:53 »

This sounds like a 'watchdog' timer on the server side that enforces an overall maximum time on a file transfer.  It if hasn't completed in that time the connection is closed, even if data is still being transferred.  I don't know about such a setting, just guessing. 

Of course someone on the microstock site would have to get involved but if it's just a configuration setting, it would be easy to change.



Ignore | Logged


Dreamframer



« Reply #8 on: May 07, 2009, 05:08 »

Did you try to disable multi-thread in CuteFTP? I had to do it because of Crestock.


Ignore | Logged


.shock



« Reply #9 on: June 27, 2009, 17:47 »

I have same problem, is it there some working solution? Anyone know how to disable multi-thread option  in cuteftp?


Ignore | Logged


Envateo Photo ToolsMicrostock Insider
Pages: [1] Print 
« previous next »
Jump to:  


Related Topics
Subject Started by Replies Views Last post
FTP Upload Issues
Off Topic
beisea 3 993 Last post September 24, 2006, 21:30
by beisea
Upload to Stock sites
General Stock Discussion
redfig 2 913 Last post December 03, 2007, 01:18
by redfig
FTP upload issues on most Stock Sites
Software - General
dnavarrojr 0 492 Last post May 06, 2009, 11:01
by dnavarrojr
how to upload images to multiple stock sites?
General Stock Discussion
JekylHyde 3 995 Last post September 30, 2009, 07:20
by Stu49
Copyright Issues On Historical Sites?
iStockPhoto.com
123XXX 10 910 Last post October 13, 2010, 14:24
by traveler1116

TinyPortal v1.0.5 beta 1© Bloc