General Category => Feature Requests => Topic started by: simonmar on February 10, 2008, 05:08:18 AM

Title: Preserve modification times?
Post by: simonmar on February 10, 2008, 05:08:18 AM
I've been looking for a way to use S3 to backup my Linksys NSLU2, and it seems like s3sync will be the most workable solution.  I have it working, but encountered one problem with using s3sync for backups: it doesn't preserve the modification time when files are restored.  So if I need to restore from the S3 backup, I lose all the modification times on my files, which are quite important to me.

This seems slightly strange, given that other metadata (owner, permissions) are preserved.  So have I missed something?  Is it hard to add?  Should I add it to the feature requests?

One other thing: I found the SSL_CERT_FILE trick from the README didn't work for me, perhaps because I'm using EU buckets and it needs a different cert.  I could use s3cmd, but tying to do an s3sync failed.  So I grabbed the certs from another Linux box, basically the contents of /etc/ssl/certs and /usr/share/ca-certificates, pointed SSL_CERT_DIR at /etc/ssl/certs, and this worked fine.

Title: Re: Preserve modification times?
Post by: jh on February 12, 2008, 08:17:22 AM
As a workaround, you can tar the files and s3sync the tarballs.  This also lets you compress and encrypt the data.

Title: Re: Preserve modification times?
Post by: Cato on April 15, 2008, 01:28:41 PM
I'd also recommend looking at DAR, an open source tool that is included in some distros including Ubuntu - it's a bit like tar but more powerful, e.g. it can compress and encrypt block-by-block, meaning that if you hit a bad disk block you would only lose one file (may not be a problem with Amazon S3 but you could hit this on a local server).  It's also got some nice features for differential and incremental backups, so you could run a single DAR base backup on installing a new system (or maybe every month) and then just do differentials (relative to base) or incrementals (relative to last incremental).

DAR also stores extended attributes in case those are important to you, and can run on almost any Linux box as it has few dependencies.

Title: Re: Preserve modification times?
Post by: AltJ on May 29, 2008, 06:28:53 PM
+1  I'd love to see mod times preserved too.

Title: Re: Preserve modification times?
Post by: pgiberson on June 03, 2008, 06:37:53 PM
I'm trying to accomplish the same thing, however I know 0 ruby so I'm having some trouble.

I have been able to add meta data to attach to my s3 files however I can't seem how to get it from S3 back to my local files and run a touch -d command...

Around line 506
Add beneath: meta['symlink'] = 'true' if fromNode.symlink?
meta['date'] = fromNode.date.to_s

When I get the headers on my files I see:
< HTTP/1.1 200 OK
< Date: Tue, 03 Jun 2008 23:26:48 GMT
< x-amz-meta-date: Sun Jun 03 23:34:26 UTC 2007
< x-amz-meta-permissions: 33204
< Last-Modified: Tue, 03 Jun 2008 23:06:48 GMT
< Content-Type:
< Content-Length: 3
< Server: AmazonS3

However I'm not sure how to get this info back and actually use it to modify my local files...

Any ideas?

Title: Re: Preserve modification times?
Post by: pgiberson on June 04, 2008, 05:48:07 PM
I've got it to work for what I need, it is probably really bad code but it works... Here it is, I would use this at your own risk. It uses touch to update the time on the file based on the meta data, if the meta data isn't there then it uses the Last Modified data from Amazon.

In s3sync.rb

               def symlink?()
                        unless @result
                                @result = S3sync.S3try(:head, @bucket, @path)
                        debug("symlink value is:  #{@result.object.metadata['symlink']}")
                        @result.object.metadata['symlink'] == 'true'
                def date2
                         unless @result
                                @result = S3sync.S3try(:head, @bucket, @path)
                        debug("date could be:  #{@result.object.metadata['date']}")


                            s3o = S3::S3Object.new(theStream, meta)
                            headers = {'Content-Length' => (fromNode.size.respond_to?(:nonzero?) ? fromNode.size.to_s : '0')}


                             headers['x-amz-meta-date'] = fromNode.date.to_s

I wasn't sure the proper place to put this, but I grouped it with the chmod/chown because it was modifying the file so I figured it was a good place to stick it.

                                # clean up if the temp file is still there (as for links)
                                File.unlink(fName) if File.exist?(fName)
                                # update permissions
                                linkCommand = fromNode.symlink? ? 'l' : ''
                                        if fromNode.date2.to_s == ""
                                                thedate = fromNode.date.to_s
                                                thedate = fromNode.date2.to_s
                                        file = @path
                                        `touch -d "#{thedate}" #{file}`