backing up large amounts of data

[frankholdem]

I've been trying to use s3sync to backup about 100 GB of data from a server to s3. Here are some observations so far:

- in my case s3sync seems to transfer data at roughly 1 GB per hour but usually runs out of retries within a few hours (3-12 hours). I'm
using the SSL option. So every few hours I have to restart the process to deal with running out of retries. Sometimes the reported error
is relating to SSL, sometimes its "Broken pipe: Broken pipe" but most times it is "500 Internal Server Error" just runs out of retries.

- I also tried using rsync from my server to a small EC2 instance and was able to achieve about 10 GB per hour. From EC2 I then used s3sync to transfer data and was able to transfer at roughly 15 GB per hour.

Conclusion: If I want to do an initial dump  of 100 GB from my server to s3 it will take about 4 days and I will have to restart the process several times due to s3sync running out of retries. If on the other hand I rsync to EC2 then s3sync from EC2 to s3 the whole process will take about 17 hours.

Are my observations consistent with what everyone else is experiencing?

I'm using version 1.2.3 on a FreeBSD 6.x system with ruby 1.8.6



- cheers
Frank

[ferrix]

# of retries is settable. 

Some ppl have reported improved tx rates running several copies at once (on different dirs obviously)

There is probably a bug in the connection code where we should reset the connection more forcefully than currently does.

Lots of needed changes, no time lately to do them.

[maelcum]

Are my observations consistent with what everyone else is experiencing?

No. Not at all. s3sync has been most stable for me. I haven't had a single retry in about 4 month of continuous transfer, in addition to a very constraint environment.

See here: http://s3sync.net/forum/index.php?topic=55.0

This was an older version of s3sync (1.1.4, if I remember correctly), though I haven't had any problems with the newer versions either. And yes, the script is still running once a week (as it takes about a day to upload anything new).

maelcum

[matchinthebox]

Frank,

How are you getting 16GB/Hour from EC2 to S3?   I'm trying the process now, and I seem to be averaging 10mb/min or 600mb/hour.

If I can get 15gb/hour, I will be thrilled!

BTW:  I'm using a "Large" instance on EC2, if that makes a difference.

Cheers
mm