S3Sync.net

General Category => General Discussion => Topic started by: treed on February 24, 2007, 08:48:00 PM



Title: A small feature request...
Post by: treed on February 24, 2007, 08:48:00 PM
First, thanks for s3sync! It is awesome. I tried a number of other of the tools to sync my data with s3 and s3sync is by far the most useful. I am holding out some hope for s3fs-fuse but the developer doesn't seem too serious about it so I don't want to rely on it yet.

So far the only thing I could ask for of s3sync is a somewhat more verbose -v and a --progress option like rsync has. I have a relatively slow connection and it is taking forever to upload some files. Sometimes I wonder if it is still working. A more verbose -v (for example, telling me that a certain file has not changed and does not need updating like rsync has) and a progress bar to show me the progress of the current file being  uploaded perhaps along with a kb/s reading would be a nice touch and make these slow uploads a bit more tolerable.

The ability to sync only chunks of files that have changed would be nice but if I understand the protocol correctly (I have read the REST and SOAP S3 specs) you cannot upload only a section of a file and instead have to upload the whole file. I suppose s3sync could upload files in smaller chunks and only re-upload the chunks that change and keep track of which chunks belong together in what order via metadata but that would complicate the code and make it impossible to retrieve those files with any other tool which is definitely handy in case someday I need my data but s3sync is not available.


Title: Re: A small feature request...
Post by: ferrix on February 25, 2007, 05:34:20 PM
--progress was a neat challenge.  And it only took me a couple hours  ;D

I'm not adding the no-op comparisons to -v.. I want to keep that for when there's actually something that "needs doing".  If you are really desperate for more info, you could either
  • Use the --deubg flag
  • Patch the comparator to change some of the '--debug' checks and debug() calls to be like the '--verbose' checks instead
NB: debug() is a simple wrapper function so I didn't have to keep typing the whole
$stderr.puts "blah blah" if $S3syncOptions['--debug']
each time.


Title: Re: A small feature request...
Post by: rondari on March 10, 2007, 09:01:26 PM
--progress option is very nice. This kind of thing is missing in a lot of the other S3 libs ... good one