Preserve modification times?

[simonmar]

I've been looking for a way to use S3 to backup my Linksys NSLU2, and it seems like s3sync will be the most workable solution.  I have it working, but encountered one problem with using s3sync for backups: it doesn't preserve the modification time when files are restored.  So if I need to restore from the S3 backup, I lose all the modification times on my files, which are quite important to me.

This seems slightly strange, given that other metadata (owner, permissions) are preserved.  So have I missed something?  Is it hard to add?  Should I add it to the feature requests?

One other thing: I found the SSL_CERT_FILE trick from the README didn't work for me, perhaps because I'm using EU buckets and it needs a different cert.  I could use s3cmd, but tying to do an s3sync failed.  So I grabbed the certs from another Linux box, basically the contents of /etc/ssl/certs and /usr/share/ca-certificates, pointed SSL_CERT_DIR at /etc/ssl/certs, and this worked fine.

[jh]

As a workaround, you can tar the files and s3sync the tarballs.  This also lets you compress and encrypt the data.

[Cato]

I'd also recommend looking at DAR, an open source tool that is included in some distros including Ubuntu - it's a bit like tar but more powerful, e.g. it can compress and encrypt block-by-block, meaning that if you hit a bad disk block you would only lose one file (may not be a problem with Amazon S3 but you could hit this on a local server).  It's also got some nice features for differential and incremental backups, so you could run a single DAR base backup on installing a new system (or maybe every month) and then just do differentials (relative to base) or incrementals (relative to last incremental).

DAR also stores extended attributes in case those are important to you, and can run on almost any Linux box as it has few dependencies.

[AltJ]

+1  I'd love to see mod times preserved too.

[pgiberson]

I'm trying to accomplish the same thing, however I know 0 ruby so I'm having some trouble.

I have been able to add meta data to attach to my s3 files however I can't seem how to get it from S3 back to my local files and run a touch -d command...

Code:

Around line 506
-----------------
Add beneath: meta['symlink'] = 'true' if fromNode.symlink?
meta['date'] = fromNode.date.to_s

When I get the headers on my files I see:
< HTTP/1.1 200 OK
< Date: Tue, 03 Jun 2008 23:26:48 GMT
< x-amz-meta-date: Sun Jun 03 23:34:26 UTC 2007
< x-amz-meta-permissions: 33204
< Last-Modified: Tue, 03 Jun 2008 23:06:48 GMT
< Content-Type:
< Content-Length: 3
< Server: AmazonS3

However I'm not sure how to get this info back and actually use it to modify my local files...

Any ideas?

[pgiberson]

I've got it to work for what I need, it is probably really bad code but it works... Here it is, I would use this at your own risk. It uses touch to update the time on the file based on the meta data, if the meta data isn't there then it uses the Last Modified data from Amazon.


In s3sync.rb

Below:
               def symlink?()
                        unless @result
                                @result = S3sync.S3try(:head, @bucket, @path)
                        end
                        debug("symlink value is:  #{@result.object.metadata['symlink']}")
                        @result.object.metadata['symlink'] == 'true'
                end
Add:
                def date2
                         unless @result
                                @result = S3sync.S3try(:head, @bucket, @path)
                        end
                        debug("date could be:  #{@result.object.metadata['date']}")
                        @result.object.metadata['date'].to_s
                end

Below:

                            s3o = S3::S3Object.new(theStream, meta)
                            debug(@path)
                            headers = {'Content-Length' => (fromNode.size.respond_to?(:nonzero?) ? fromNode.size.to_s : '0')}


Add:

                             headers['x-amz-meta-date'] = fromNode.date.to_s


I wasn't sure the proper place to put this, but I grouped it with the chmod/chown because it was modifying the file so I figured it was a good place to stick it.

Below:
                                # clean up if the temp file is still there (as for links)
                                File.unlink(fName) if File.exist?(fName)
                                # update permissions
                                linkCommand = fromNode.symlink? ? 'l' : ''
                                begin
Add:
                                       
                                        if fromNode.date2.to_s == ""
                                                thedate = fromNode.date.to_s
                                        else
                                                thedate = fromNode.date2.to_s
                                        end
                                        file = @path
                                        `touch -d "#{thedate}" #{file}`