all right all right.. it *is* the weekend, I suppose this counts as recreation ye?

cm_gui: I'm sorry you're having trouble, but posting the same request in another way without understanding the first response isn't going to get us anywhere.

As we have repeatedly said, the object size limit, as well as the 2GB bug, is for an individual S3 node (in this case, files), not collections of them (folders).  s3sync does make a node representing the folder, but it is simply a small placeholder to record the permissions etc.  It will in no way impact size limitations.

I haven't tested that.  I think it's clear from the above that 1) it's something in ruby but 2) not in my code per se, and 3) it doesn't happen for everyone.

I don't see anything in those results that point to s3sync doing dumb looping or etc.

However both for you and for lfh's results (he is not using my software), the Thread#kill and Timeout.timeout seem to be taking over 30% of the time if I'm reading it right.

What I'm not sure of, is whether these results aggregate inner calls to the other inner functions, or whether the time is spent *just* and *only* executing code native to that function.

I would assume the former, which means this result isn't all that informative.  For example it might be spending time in "timeout" waiting for the network result to come back, etc.

I am *not* an expert in ruby by any stretch, so I'm probably not the most qualified to pursue these details.

Yes slashes are magically handled right.  In fact with s3sync, if you talk to it with DOS-style back slashes, it gets mightily confused.

Ruby comes with tracing and debugging available, though I haven't used it too much myself.

It would certainly be helpful if we found a way to get performance boosted substantially!  I find it equally likely that the problem is within the Ruby library code somewhere, but if we can learn where we might still be able to patch around the inefficiency.

It could also be something such as the default read/write chunk size might be too small.

Added nifty option --progress
It displays a status line during long transfers showing bytes transferred, transfer speed, and (if known) percentage of file completed.  (This is not known when, for example, doing a "get" in s3cmd, since we don't do a special "head" command just to learn the file size, and the meta data from S3.rb isn't available until the operation completes)

--progress was a neat challenge.  And it only took me a couple hours 

I'm not adding the no-op comparisons to -v.. I want to keep that for when there's actually something that "needs doing".  If you are really desperate for more info, you could either

• Use the --deubg flag
• Patch the comparator to change some of the '--debug' checks and debug() calls to be like the '--verbose' checks instead

NB: debug() is a simple wrapper function so I didn't have to keep typing the whole
$stderr.puts "blah blah" if $S3syncOptions['--debug']
each time.

OK so .. I know this is going to start a trend of ".0 versions are unstable" right?

But there was a stupid bug I introduced in the path handling changes, and I fixed it.  Definitely use 1.1.1 instead of 1.1.0.  I still advise caution when upgrading from the 1.0.x line in a production system.

Augh.. am I ever an idiot 

Fix coming.

I understand what you mean, however I would like to maintain the distinction between environment variables for general invariant configuration information, and options for instructions for the current command invocation.

For example, it's very possible that despite having the SSL env variable set, someone might want to do some non-ssl S3 transfers.

Thanks for the suggestion 

Will take a look!

What you're asking for is to reverse the way the system behaves, so that your way will become the default?  I don't understand why don't you juse use -s instead of changing the code?

Meh.. I don't have any files >2gb myself.. or else I long since would have been complaining and/or figuring out some way around it.

As you all may notice, the best way to motivate me is .. when I have the problem myself 

I already check the size before the etag and skip etag if the size is different.  But the problem is that 99% of the time the size is the same, so the etag check has to go ahead.  Of course no one would claim that "same size" is good enough to know that the files are the same.

The point of my comment on modified time is not to restore timestamps... but rather to use them as a method of determining whether to sync *in lieu* of etag checking.  This way we would always be able to use meta data locally to check, rather than md5'ing the file.

I wouldn't make this the default, or take the old behavior away.  Just some low-hanging fruit to speed things up especially on my slow ass windows XP machine.