ok then, now you can claim it is self-healing software, another required buzzword checkoff ;-).

the way I do it is mark directories and symlinks in the metadata, then when I do the HEAD call I know I can restore them without doing a GET, just by using the metadata content.  as previously mentioned, I punt on hardlinks (I PUT them once for each link, i.e. break the links).  I  do the HEAD calls first to check the etag so as not to retrieve something I already have.  my typical file is around 2-3 MB, so the overhead of the HEAD is in the noise.

since s3 is really a name/value database, not a filesystem, the guarantee of atomicity would make it very difficult to resume.  think about writing over an existing key. you start transferring the bytes and lose the connection at some point. how would they know if you just got tired of waiting or if you wanted to resume? how long woud they keep partials?  suppose you try to resume from two connections?  the protocol would require the clients to query the server and be able to understand where to start, which doesn't fit in very well with REST hey? I can understand how it works when I download with ftp, the file on the server end hasn't changed so a smart client can tell the server, "hey, start at byte 233333  and give it to me again".  if the file meanwhile has changed though, how does the client know that and how would the server know if it had changed since the client last asked?  if the client dies ftp doesn't keep any state, so the client and server have to be making some assumptions to allow the resume to happen.  and rsync doesn't allow resume at all right? it just starts in on the diffs again and resends what it has to?

in a nutshell I think if they claim to support resume (I haven't seen the claim) I don't believe them.  where is the explanation of the protocol?

rt, 500s are on their side, but people constantly complain about timeouts and drops, and with those it is far from clear cut who is at fault.

for starters, giving the exact command line your are running would probably help in diagnosis.

"Usually something goes wrong (on the amazon side I would presume) and the connection gets dropped. "

I don't see why you should assume it's on the aws side.  evidence?  even granting that you can reliably upload to some other site the network path could be quite different.

yes, the perversity of users is well known to be unbounded, that's why big projects use betas (of course this doesn't include S3, they just roll it out and pray) 

you didn't up the version? shame on you! take yourself to the woodshed!

missing one of these points out how difficult it is be sure all your code is working.  after a long series of changes, all of which worked as far as I could tell, I tried listing an empty bucket. boing! wrong number of arguments error in a logging function!  I thought I'd covered them all by manually editing the code and causing the error side of conditionals to execute, but I was wrong about that.

man, timing out while writing a long reply sucks 

it seems to me there are several cases:

1) user attempts to retrieve a file or directory whose stored name is illegal on the local system;  File.new() or FileUtils.mkdir_p raises Errno::EINVAL; I suggest catching it, reporting it with a warning, and continuing (note the directory case may cause a cascade of failures)

2) user attempts to retrieve a file whose type is not supported on the local system (e.g. symlink); File.symlink() raises NotImplementedError; I suggest catch/report/continue, as above

3) user attempts to restore a file when a local object by the same name but wrong type exists; depending how you do the check you might get Errno::EEXIST or you might find out in advance so no exception happens; report and continue

4) user attempts to archive a file that you don't restore correctly, e.g. hardlink;  I suggest noting that it will be copied rather than saved as a link (and if users complain, fine, let them suggest how it could be handled, as we have discussed it isn't a simple problem)

what I don't suggest doing is trying to figure out in advance if a file is going to be illegal or unsupported somewhere else...after all, the file exists on the local machine and that may be the only place the user cares about, and also there is the chance that ruby will support something like symlinks on windows in the future. 

of course if the user tries to archive a file that can't be restored locally that's a different issue.  I only allow ordinary files, symlinks and directories, so if the user tries to archive something like /dev/zero, a socket, a FIFO, etc, I just say "so sorry" and go on.

I use some wrapper classes to hide the differences between various file types so my code to catch these errors is relatively centralized.  the case of files exists locally with wrong type I just check before attempting any restore.

ps:

one more thing I didn't attempt is windows ntfs ACLs and SELinux security contexts. see http://www.phptr.com/articles/article.asp?p=606586&rl=1 for a little idea of how deep the swamp is.  my assumption is users sophisticated enough to care about these will use a fancier solution than I could provide, and of course there is the little problem that I don't know if ruby even supports reading/writing the extended attributes involved.

heh, you are limiting yourself unnecessarily ;-). 

example:  I want to share a bunch of pics and also back them up.  I'm behind a NAT router and so is the sharee.  sure, I can set up ways to tunnel, mess with the routers on both ends, etc, but I'm lazy.  I put the files on s3 (backup function done already then), and restore them on the other machine, no problem with firewalls, NAT, etc, done.  the first time it's just cp, cp, cp.  but after that?  maybe I don't want to be careful about what's been sent and what hasn't.  voila, no problem with s3sync...sync up to s3 and then sync down at the other end.  same thing goes with any set of files, it needn't be pics.

example:  I have some files I want to be kept very very safe, but I want somebody else to be able to access them if needed, for instance if I die.  again I set up a script to do the sync down and another to decrypt the files with a key provided when the time comes.  the sharee has an amazon account and I set up the ACLs accordingly.  I don't worry about security because the files are encrypted, and I can change the contents of the bag anytime I please.  again the A->B->C setup is far easier for me to manage than trying to set up a direct A->C. 

now granted, the above takes some savvy to set up too, but all the smarts can be at my end, I set up the other end with a batch file to do the sync, say "double click here" and we are done.  rsync direct A<->B would be a nightmare to set up by comparison, and there is always the possibility that the ISP's firewall rules won't let it pass.  it avoids simpler failures too, for example S3 is up 24/7 and one disk crash isn't going to put them out of business (we now pause to say goodbye to the svn server I had on the spare box...).

in other words I think rsync + REST opens possibilities that rsync alone doesn't.

I'm surprised you haven't run into it already.  there are lots of reasons to use something like rsync, and one of them is to do things like replication of a tree from one system to another; this comes up with trees of static web content.  of course people don't usually have perverse file names on purpose, but there are obvious ones, e.g. mm:dd:yyyy.log, that work fine on unix-like systems but not windows.

something similar to symlinks (called by some other, much more baroque name in typical ms fashion) also exists in NTFS, in fact there are two different things: shell links (takes two files per link) and "reparse points" (much closer to symlinks), but AFAIK ruby doesn't support them nor the alternate streams.

mine are 1.8.5 also.  it's possible there were fixes in the http lib, although it is hard to imagine what would affect what is basically a loop writing into a socket.

here is a cute thing I did run into:  as everybody knows, the filesystem illegal char list is different between windows and linux.  AFAIK only \0 and '/' are illegal for files on linux.  so I have a file in my test bucket named
    'data/ x" y '
i.e. has a trailing space and a double-quote.  the double-quote is illegal on windows, which is reported by throwing an Errno::EINVAL if you try to open it.  if you don't already catch that you might want to take a look at your code.  symlinks are another issue, you get a NotImplementedError if you call File.symlink().  annoyingly, File.respond_to? returns true.

of course total time *would* be of interest if it didn't match up pretty well with available bandwidth ;-).

right, the time when running the profiler is meaningless.  the profiler slows things down enough by itself that it could mask other issues, e.g. disk or network waits.  still, the profiler gives the raw counts which can be quite useful.

were you testing by grabbing the 20M link I provided?  if yes, then you are taking 50% or so more time than I do to get it.  my DSL connection is the bottleneck as far as I can tell, i.e. my download speed seems to be about the same no matter what tool I use.

brad:  what OS are you running?