S3Sync.net
February 02, 2014, 01:38:39 PM *
Welcome, Guest. Please login or register.

Login with username, password and session length
 
   Home   Help Search Login Register  
Pages: [1]
  Print  
Author Topic: backing up large amounts of data  (Read 3862 times)
frankholdem
Newbie
*
Posts: 9


View Profile
« on: December 23, 2007, 06:51:03 PM »

I've been trying to use s3sync to backup about 100 GB of data from a server to s3. Here are some observations so far:

- in my case s3sync seems to transfer data at roughly 1 GB per hour but usually runs out of retries within a few hours (3-12 hours). I'm
using the SSL option. So every few hours I have to restart the process to deal with running out of retries. Sometimes the reported error
is relating to SSL, sometimes its "Broken pipe: Broken pipe" but most times it is "500 Internal Server Error" just runs out of retries.

- I also tried using rsync from my server to a small EC2 instance and was able to achieve about 10 GB per hour. From EC2 I then used s3sync to transfer data and was able to transfer at roughly 15 GB per hour.

Conclusion: If I want to do an initial dump  of 100 GB from my server to s3 it will take about 4 days and I will have to restart the process several times due to s3sync running out of retries. If on the other hand I rsync to EC2 then s3sync from EC2 to s3 the whole process will take about 17 hours.

Are my observations consistent with what everyone else is experiencing?

I'm using version 1.2.3 on a FreeBSD 6.x system with ruby 1.8.6



- cheers
Frank

« Last Edit: December 23, 2007, 08:42:00 PM by frankholdem » Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #1 on: December 23, 2007, 09:12:44 PM »

# of retries is settable. 

Some ppl have reported improved tx rates running several copies at once (on different dirs obviously)

There is probably a bug in the connection code where we should reset the connection more forcefully than currently does.

Lots of needed changes, no time lately to do them.
Logged
maelcum
Newbie
*
Posts: 43



View Profile
« Reply #2 on: January 06, 2008, 01:45:22 PM »

Are my observations consistent with what everyone else is experiencing?

No. Not at all. s3sync has been most stable for me. I haven't had a single retry in about 4 month of continuous transfer, in addition to a very constraint environment.

See here: http://s3sync.net/forum/index.php?topic=55.0

This was an older version of s3sync (1.1.4, if I remember correctly), though I haven't had any problems with the newer versions either. And yes, the script is still running once a week (as it takes about a day to upload anything new).

maelcum
Logged
matchinthebox
Newbie
*
Posts: 2


View Profile
« Reply #3 on: January 14, 2008, 06:00:50 PM »

Frank,

How are you getting 16GB/Hour from EC2 to S3?   I'm trying the process now, and I seem to be averaging 10mb/min or 600mb/hour.

If I can get 15gb/hour, I will be thrilled!

BTW:  I'm using a "Large" instance on EC2, if that makes a difference.

Cheers
mm
Logged
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2013, Simple Machines Valid XHTML 1.0! Valid CSS!