I don't recall the exact limit, but it is small, maybe 2 minutes or less.  try using ntp to keep your clocks accurate.

I don't follow you here...a filename in common linux filesystems is just a sequence of bytes not including '\0' or '/'.  these bytes are displayed by various tools according to the current locale, but that has no effect on the underlying name.

s3 isn't the problem, according to AWS:

"We support from U+0001 to U+10FFFF (null character is encodedlikethis: %C0%80). This range is supported for GETs and PUTs, but someXMLparsers might choke on LIST if there are any unpritable charactersin the keys. (We return entity references like  forthesecharacters, but some parsers choke on these). "

there is a good reason why "some parsers choke", because most chars below #x20 are not legal in xml 1.0, which is what a list bucket request returns.  however, it turns out the ruby REXML parser will correctly parse all those illegal chars even though it isn't supposed, so you can list, get and put these chars as well as the legal ones.

I have testing get/put files with the illegal chars in them as well as files with umlauts with no problem.

shorter answer:  either s3sync is doing something bad (I doubt it if it is using REXML) or...

export RUBYLIB=/path/to/your/libraries

e.g. if
  require 'blah'

and /path/to/blah then

export RUBYLIB=/path/to

is your clock right?  there have been a lot of issues over at s3 caused by inaccurate clocks.

http://developer.amazonwebservices.com/connect/message.jspa?messageID=60477#60477

lfh

http://amazon.rubyforge.org/

the code appears to be idiomatic rails code, thus simple to pick up for rails riders.  streaming up/down is there already, but I didn't see anything to deal with directories or symlinks, and nothing to list prefixes.

fyi, aws thinks the 2GiB limit bug is fixed.

this is the first thing that popped up by googling
http://www.google.com/search?q=ruby+1.8.4+mac&start=0&ie=utf-8&oe=utf-8&client=firefox-a&rls=org.mozilla:en-US:official

 http://wincent.com/knowledge-base/Building_and_installing_Ruby_1.8.4_on_Mac_OS_X_Tiger
it looks pretty simple to me, but I'm used to building stuff on linux.

I agree about 1.8.2 being the wrong thing to use, but isn't streaming something you had to add in by modifying Net::HTTP or one of its subclasses?  I did it like this in S3.rb instead:
        if req.request_body_permitted?  # post/put
          if data.respond_to?(:read)    # file or file-like?

            # setting body_stream and calling http.request with the
            # second argument nil causes the put to stream from the
            # file instead of having to load the whole thing in memory

            req.body_stream = data
            result = @http.request(req,nil)
            # ...

tar may be able to continue, but what does it actually do, and in what circumstances?  I can't see that there is one set of rules that will result in the desired result, since the desired result would vary according to the user.   think about a few possibilities: 1) another process opens the file for writing and truncates it to zero; 2) another process appends to the file; 3) another process writes a block into the middle of the file;  4) another process deletes the file. 

I wouldn't make the assumption that tar is simple, after all it's been around for more than 20 years and even today there is active development on some versions of it.  and don't forget, tar and ruby have exactly the same set of system calls available to them, so there is no magic going on here.

as far as S3 is concerned, one issue right off the bat is you need to send the content-length header.  what should it be for a file that is continuing to grow?  do you stop after you stat it the first time?  no matter what happens, eventually what you archive will not agree with what is on the drive, is this acceptable?   for a growing file the real rsync sends however many bytes it initially stats; for a shrinking file I don't  know what happens (it will notice the byte count sent is wrong, but I don't know what happens on the server side).

there *are* a few things s3sync should not do, e.g. hang or throw an unhandled exception ;-).

greg:  there is some conditional code in S3.rb that tests for ruby version < 1.8.3, but I forget if that's what you are using.

linux:
  man 2 flock
  man 2 fcntl

I don't recall on win32.  the short answer is it depends on the OS and on your current privileges.

suppose you could lock a file whenever you wanted: think about what would happen to other processes that were trying to write to it if your network upload speed suddenly shrank to 300 baud. :-(

I don't know the internals, but presumably it is making up a list and then processing it?  if it processed a file after opening it, it wouldn't matter (on unix-like systems) if you deleted the file, it would back it up anyway because the file doesn't go away until the ref count is zero.  of course it shouldn't barf if opening a file fails ;-).

I deal with this by checking File.ftype; if it isn't "file", "directory" or "link" I skip it as unsupported.  obviously it would take a while to archive /dev/zero, and then there *is* the issue of the length...

key content larger than 2GB is currently not supported by s3.  this issue has been around for nearly a year and aws just keeps saying "we are working on it".