|
Hi folks,
I ran into the "Broken pipe" problem. Trying to upload 20 GB of ~1 MB pictures constantly failed after a while with "broken pipe" and the retries simply counted down to 0 with no effect.
A bit of debugging (with no ruby skills) turned out that s3try.rb seems to be the place where things go wrong.
The method/function "S3try" handles the connection and the actual transfer of data (I think...).
Looking at the code I noticed that the retries are handled in a while loop.
However the code which seems to create a connection to S3 is outside this loop at the beginning of S3try.
If a connection error (e.g. Broken Pipe) happens the connection does not get reset/rebuild and all other retries of course fail too. It doesn't matter how high you set the retry counter...
So I thought it might be good idea to include the connection code inside the loop and let it only run if needed (e.g. broken pipe).
All I did was move the line
while $S3syncRetriesLeft > 0 do
up just below the S3try declaration and added a line which sets forceRetry to false.
The if statement is modified to also let a new connection happen if something bad happened before (e.g. a broken pipe resulted in forceRetry=true)
So the S3try method now starts like this:
def S3sync.S3try(command, bucket, *args)
forceRetry=false
while $S3syncRetriesLeft > 0 do
if(not $S3syncHttp or (bucket != $S3syncLastBucket) or forceRetry==true)
$stderr.puts "Creating new connection" if $S3syncOptions['--debug']
$S3syncLastBucket = bucket
$S3syncHttp = $S3syncConnection.make_http(bucket)
end
I'm uploading for some time now, 3 broken pipes so far but connection was re-created and upload continues.
So at least for me this problem is no longer - I hope....
|
Show Posts
