How do I do incremantals with S3Sync?

[scotch]

Hello,

I just got S3Sync up and running on a Linux server and just did a full backup of files to my S3 bucket. But I can't seem to find anything in the documentation about doing incrementals. Could someone steer me toward any info on this?

Thanks,
scotch

[ferrix]

Like rsync, it detects whether each item needs to be changed.  By default it uses the md5 values.  Sometimes that's too expensive, so md5 checking can be turned off and then it will rely on file size and (if I recall correctly) upload time.

[scotch]

So by default it essentially does incrementals by checking to see if files have been changed and then backing those up? If that's the case, then how do I do a full backup? I was hoping to have once script for daily incrementals and one script for full backups every friday night, so something like:

ruby s3sync.rb -r -s -v "--incremental_switch" /var/www/ my_s3_bucket_incrementals:www_inc
ruby s3sync.rb -r -s -v "--full_switch" /var/www/ my_s3_bucket_fulls:www__full

Thanks for your help.

[ferrix]

By nature sync overwrites older stuff with the newest changes.  The nature of "incremental backup" is to preserve the older state AND save the new stuff off to the side. 

The point being, you want to minimize storage but be able to get access to 7 days of data.  You can't have both with a simple sync-like solution.  I'd recommend 7 different save locations, rotating each day which one you write to.  Otherwise you will only have the latest day of backup.

[scotch]

I'd recommend 7 different save locations, rotating each day which one you write to.

Ah, that's a great suggestion. I'll just have a cron job for each day, backing up to that day's bucket. Thanks very much.

[ferrix]

Yep that's how I do it.

Once I rewrite s3sync completely and take advantage of copy/rename, we can do it in a more rsync-like way that doesn't require uploading every change 7 times.