I have motivated the development of an OpenBSD port of s3sync, intended to be suitable for the OpenBSD ports repository.

I've attached the port to this message.

I submitted this to ports@openbsd.org, but it hasn't been accepted/committed yet, how this might actually happen is a bit of a mystery to me.  I've tested this port on OpenBSD 4.3 (stable).

In any case, I would certainly appreciate people testing this port, and letting me know if there are any issues.

Don

OK, my script seems to detect the broken pipe situation via the return code, re-try, and last night, the re-try succeeded.

Here is the tail end of the first attempt:

1 retries left, sleeping for 30 seconds
Broken pipe: Broken pipe
0 retries left, sleeping for 30 seconds
Ran out of retries; operations did not complete!
Ran out of retries; operations did not complete!
/usr/local/bin/s3sync.rb:286:in `s3TreeRecurse': undefined method `http_response' for nil:NilClass (NoMethodError)
   from /usr/local/bin/s3sync.rb:333:in `s3TreeRecurse'
   from /usr/local/bin/s3sync.rb:314:in `each'
   from /usr/local/bin/s3sync.rb:314:in `s3TreeRecurse'
   from /usr/local/bin/s3sync.rb:333:in `s3TreeRecurse'
   from /usr/local/bin/s3sync.rb:314:in `each'
   from /usr/local/bin/s3sync.rb:314:in `s3TreeRecurse'
   from /usr/local/bin/s3sync.rb:346:in `main'
   from /usr/local/bin/thread_generator.rb:79:in `call'
   from /usr/local/bin/thread_generator.rb:79:in `initialize'
   from /usr/local/bin/thread_generator.rb:76:in `new'
   from /usr/local/bin/thread_generator.rb:76:in `initialize'
   from /usr/local/bin/s3sync.rb:267:in `new'
   from /usr/local/bin/s3sync.rb:267:in `main'
   from /usr/local/bin/s3sync.rb:735
Update node awstats/awstats072008.mail2.clark-communications.com.txt
Create node backup/db_2008-07-15.sql.gz

Return code from s3sync: 1

As the return code was non-zero, my script slept for 5 minutes, and then retried the s3sync command, the second time it worked.

So, I'll continue to monitor my script, but so far, this is a major improvement, my backup retries until it succeeds.

I continue to get the broken pipe message about every other time I run s3sync.

I added the code to my shell script to capture the return code.

When the s3sync runs to completion, it does seem to return 0.

When s3sync gets caught in a long series of broken pipe timeouts, and I manually kill it the next the next morning, the return code is 137.

So, it would be ideal if I didn't get stuck with the broken pipe issue so often, but until I find the root cause of that,
my proposal is to return to a reasonable/small number of retries, and hopefully if s3sync gets to the end of retries, it will return a non-zero code, and I"ll modify my script to check the return code, and re-try s3sync later.

Any other thoughts/suggestions/insight would be most welcome.

My shell script using s3sync called by cron is moving along.

Today it ran to completion, and all was well.

Yesterday though, I got many of the following errors:

Broken pipe: Broken pipe
1023 retries left, sleeping for 30 seconds
Broken pipe: Broken pipe
1022 retries left, sleeping for 30 seconds

I am running with the --verbose option.

The backup job essentially frozen, and many hours later I manually killed it.

Any ideas what causes this to happen, or things I could do to recover from this?

In case you are wondering, I do specify a large retry value (1024), because I wanted s3sync to keep trying to get its backup done.
This was important the first time I backed up the (large) directory tree, but maybe less important now.

Second question:

Does s3sync provide any sort of return code I could use to know if it was successful?
If so, that might be a way for me to automatically determine if the backup was successful or not.

Hello,

I'm using s3sync to back up an important directory tree on one of my servers.

I'm running s3sync from cron every night, I finally got all the environment variables set correctly so it runs.

Since cron packages up the output of the apps it runs, and emails them to root, I'd like to have s3sync provide information about what
it is doing.

When I run s3sync interactively, I specify --progress, which is a great feature, but when run under cron, I don't want the percentage file xfer stats that progress provides, just the file operations

in other words, I want these msgs:
     node backup/db_2008-07-07.sql.gz
but not these msgs:
Progress: 90112b  87852b/s  2%       
Progress: 169984b  83748b/s  4%     

(how) can I get s3sync to do this?

Thanks!

Don