S3Sync.net
February 02, 2014, 01:23:33 PM *
Welcome, Guest. Please login or register.

Login with username, password and session length
 
   Home   Help Search Login Register  
Pages: [1]
  Print  
Author Topic: How do I do incremantals with S3Sync?  (Read 4387 times)
scotch
Newbie
*
Posts: 3


View Profile
« on: September 23, 2010, 12:41:35 PM »

Hello,

I just got S3Sync up and running on a Linux server and just did a full backup of files to my S3 bucket. But I can't seem to find anything in the documentation about doing incrementals. Could someone steer me toward any info on this?

Thanks,
scotch
Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #1 on: September 24, 2010, 03:55:04 PM »

Like rsync, it detects whether each item needs to be changed.  By default it uses the md5 values.  Sometimes that's too expensive, so md5 checking can be turned off and then it will rely on file size and (if I recall correctly) upload time.
Logged
scotch
Newbie
*
Posts: 3


View Profile
« Reply #2 on: October 04, 2010, 12:49:27 PM »

So by default it essentially does incrementals by checking to see if files have been changed and then backing those up? If that's the case, then how do I do a full backup? I was hoping to have once script for daily incrementals and one script for full backups every friday night, so something like:

ruby s3sync.rb -r -s -v "--incremental_switch" /var/www/ my_s3_bucket_incrementals:www_inc
ruby s3sync.rb -r -s -v "--full_switch" /var/www/ my_s3_bucket_fulls:www__full

Thanks for your help.
Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #3 on: October 05, 2010, 07:37:22 AM »

By nature sync overwrites older stuff with the newest changes.  The nature of "incremental backup" is to preserve the older state AND save the new stuff off to the side. 

The point being, you want to minimize storage but be able to get access to 7 days of data.  You can't have both with a simple sync-like solution.  I'd recommend 7 different save locations, rotating each day which one you write to.  Otherwise you will only have the latest day of backup.
Logged
scotch
Newbie
*
Posts: 3


View Profile
« Reply #4 on: October 05, 2010, 11:13:32 AM »

Quote
I'd recommend 7 different save locations, rotating each day which one you write to.

Ah, that's a great suggestion. I'll just have a cron job for each day, backing up to that day's bucket. Thanks very much.
Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #5 on: October 05, 2010, 08:30:35 PM »

Yep that's how I do it.

Once I rewrite s3sync completely and take advantage of copy/rename, we can do it in a more rsync-like way that doesn't require uploading every change 7 times.
Logged
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2013, Simple Machines Valid XHTML 1.0! Valid CSS!