S3Sync.net

My shell script using s3sync called by cron is moving along.

Today it ran to completion, and all was well.

Yesterday though, I got many of the following errors:

Broken pipe: Broken pipe
1023 retries left, sleeping for 30 seconds
Broken pipe: Broken pipe
1022 retries left, sleeping for 30 seconds

I am running with the --verbose option.

The backup job essentially frozen, and many hours later I manually killed it.

Any ideas what causes this to happen, or things I could do to recover from this?

In case you are wondering, I do specify a large retry value (1024), because I wanted s3sync to keep trying to get its backup done.
This was important the first time I backed up the (large) directory tree, but maybe less important now.

Second question:

Does s3sync provide any sort of return code I could use to know if it was successful?
If so, that might be a way for me to automatically determine if the backup was successful or not.

I continue to get the broken pipe message about every other time I run s3sync.

I added the code to my shell script to capture the return code.

When the s3sync runs to completion, it does seem to return 0.

When s3sync gets caught in a long series of broken pipe timeouts, and I manually kill it the next the next morning, the return code is 137.

So, it would be ideal if I didn't get stuck with the broken pipe issue so often, but until I find the root cause of that,
my proposal is to return to a reasonable/small number of retries, and hopefully if s3sync gets to the end of retries, it will return a non-zero code, and I"ll modify my script to check the return code, and re-try s3sync later.

Any other thoughts/suggestions/insight would be most welcome.

OK, my script seems to detect the broken pipe situation via the return code, re-try, and last night, the re-try succeeded.

Here is the tail end of the first attempt:

1 retries left, sleeping for 30 seconds
Broken pipe: Broken pipe
0 retries left, sleeping for 30 seconds
Ran out of retries; operations did not complete!
Ran out of retries; operations did not complete!
/usr/local/bin/s3sync.rb:286:in `s3TreeRecurse': undefined method `http_response' for nil:NilClass (NoMethodError)
   from /usr/local/bin/s3sync.rb:333:in `s3TreeRecurse'
   from /usr/local/bin/s3sync.rb:314:in `each'
   from /usr/local/bin/s3sync.rb:314:in `s3TreeRecurse'
   from /usr/local/bin/s3sync.rb:333:in `s3TreeRecurse'
   from /usr/local/bin/s3sync.rb:314:in `each'
   from /usr/local/bin/s3sync.rb:314:in `s3TreeRecurse'
   from /usr/local/bin/s3sync.rb:346:in `main'
   from /usr/local/bin/thread_generator.rb:79:in `call'
   from /usr/local/bin/thread_generator.rb:79:in `initialize'
   from /usr/local/bin/thread_generator.rb:76:in `new'
   from /usr/local/bin/thread_generator.rb:76:in `initialize'
   from /usr/local/bin/s3sync.rb:267:in `new'
   from /usr/local/bin/s3sync.rb:267:in `main'
   from /usr/local/bin/s3sync.rb:735
Update node awstats/awstats072008.mail2.clark-communications.com.txt
Create node backup/db_2008-07-15.sql.gz

Return code from s3sync: 1

As the return code was non-zero, my script slept for 5 minutes, and then retried the s3sync command, the second time it worked.

So, I'll continue to monitor my script, but so far, this is a major improvement, my backup retries until it succeeds.