resume upload

[rondari]

S3 doest support resume upload does it?

bummer that. ... I bet lots of people want this. I guess I should put this on the S3 forums.

[treed]

Resume would be VERY nice. For the last couple weeks I have been trying to upload a series of digital video files to S3 that I really want to have backed up. Each is around 1G in size. I would say it requires 5 attempts each to fully upload each file. Usually something goes wrong (on the amazon side I would presume) and the connection gets dropped. This is costing me a lot of money in bandwidth for the retries.

The only way to implement resume that I can think of (and also solve the 5G problem) is if s3sync would split up files bigger than, for example, 100M into 100M chunks and then maintain some metadata on which files are to be put back together. I know this violates the goal of all data being able to be downloaded and restored without the aid of s3sync but at least all you would have to do is to cat the files back together which most sysadmins can script up easily. If you name them sensibly (filename.avi-s3sync-1of5 or something like that for a wild guess) it shouldn't be that hard. It might be a good idea to make the chunk size configurable.

[ferrix]

Well .. you could always create a patch 

If you're upset about AWS charging you for failed uploads, why not complain to them?  If they add the ability to do resume, or some other form of modifying existing objects, I will jump on it as fast as all the other devs I'm sure!

The problem, as I see it, is that s3 offers this theoretical feature of large objects, but they have two enormous practical problems with it.

• The +2GB upload bug
• The fact that there's no reliable way to send a large object to them

I could break the 'sync' paradigm to make an ugly patch on these problems.  But I won't.

[lowflyinghawk]

"Usually something goes wrong (on the amazon side I would presume) and the connection gets dropped. "

I don't see why you should assume it's on the aws side.  evidence?  even granting that you can reliably upload to some other site the network path could be quite different.

[ferrix]

Well for example if you get 500 responses or some other non network-looking hiccup, it's easy to tell that it's on their side.

My larger point is that they don't provide any reliable way to do large uploads.  Resume, or some other solution, should be supported.

[lowflyinghawk]

rt, 500s are on their side, but people constantly complain about timeouts and drops, and with those it is far from clear cut who is at fault.

[ferrix]

Yeah so, let's assume those kinds of things are inevitable.  *Other* protocols that claim to support large uploads have some method to mitigate this problem built in.  S3 though, says "hey we support this" but doesn't provide any tangible way to do it.  That's not a feature imo.

[lowflyinghawk]

since s3 is really a name/value database, not a filesystem, the guarantee of atomicity would make it very difficult to resume.  think about writing over an existing key. you start transferring the bytes and lose the connection at some point. how would they know if you just got tired of waiting or if you wanted to resume? how long woud they keep partials?  suppose you try to resume from two connections?  the protocol would require the clients to query the server and be able to understand where to start, which doesn't fit in very well with REST hey? I can understand how it works when I download with ftp, the file on the server end hasn't changed so a smart client can tell the server, "hey, start at byte 233333  and give it to me again".  if the file meanwhile has changed though, how does the client know that and how would the server know if it had changed since the client last asked?  if the client dies ftp doesn't keep any state, so the client and server have to be making some assumptions to allow the resume to happen.  and rsync doesn't allow resume at all right? it just starts in on the diffs again and resends what it has to?

in a nutshell I think if they claim to support resume (I haven't seen the claim) I don't believe them.  where is the explanation of the protocol?

[ferrix]

It could be made to work I think 

Um about download resume, I don't remember for sure but I "could have sworn" that was supported.  Maybe I'm just out of my mind.