S3Sync.net
February 02, 2014, 01:27:54 PM *
Welcome, Guest. Please login or register.

Login with username, password and session length
 
  Home Help Search Login Register  
  Show Posts
Pages: [1]
1  General Category / Feature Requests / Re: Option to encrypt stored data using gpg on: December 13, 2007, 05:54:43 AM
My second try, this time by storing the unencrypted MD5 as metadata.

Files changed from official version : s3try.rb (same as try #1 above), HTTPStreaming.rb (some changes to CryptedStream class), and s3sync.rb.

On the plus side : 1) less load on the CPU as the local files do not have to be encrypted to compare MD5 with S3, this makes a difference with the prior approach when comparing identical files between S3 and local as the process could be CPU-bound instead of bandwidth bound, 2) much less changes to the official version (about 10 lines of code added to s3sync.rb).

Downside : a get headers command is necessary for each file, this slows down the process noticeably (x2 when files are both present on S3 and locally, no changes otherwise)

Configuration variables are unchanged.
2  General Category / Feature Requests / Re: Option to encrypt stored data using gpg on: December 07, 2007, 04:21:22 PM
Here's my shot at implementing encryption.

It's probably not very well coded and I'm not completly satisfied by the way it's implemented (see todo below) but it works. If somebody has time (or is brave enough) to give it a try, I'd be glad to hear his ideas to improve it.

I have changed :
  • s3try.rb (to catch decryption errors),
  • HTTPStreaming.rb (to add a CryptedStream class on the same model as the ProgressStream class),
  • s3Sync.rb (renamed here s3syncC.rb).

All other files are unchanged. It also requires the openssl and digest/sha2 ruby libraries but they 're usually bundled in the ruby package

  • Encryption is only used on the file contents; The file names, directories and symlinks are not encrypted.
  • Encryption is used when uploading files to S3 if there is a $ENCRYPTION_ALGO constant set in config.yml file (pointing to the desired openssl encryption algorithm, for example "aes-256-cbc"). Additionnaly you may set a $ENCRYPTION_KEY constant for your password, though may also type in the password at runtime if you're not comfortable in storing you password in clear in the confg file.
  • Decryption is used when downloading from S3 if the "encrypted" flag is set in the metadata (this flag is set automatically when uploaded). No other metadata is created, not even the unencrypted file md5. For this to work, s3sync.rb calculates both crypted and uncrypted md5 before assessing if a given file needs a refresh (not good - see todo). If the password is incorrect, an error is thrown and the local file (if there is one) is not overwritten.

Note that unencrypted files will be handled well too, but once you start to use encryption you cannot revert to the official s3sync.rb as it will not recognise that a file on S3 is encrypted and will update local files with encrypted data. You have been warned !

Todo :

- either store uncrypted file md5 as metadata or optimize the comparison process (with this version each file is encrypted once for md5 comparison, and possibly a second time if the file needs to be uploaded - it could cache the encrypted file -). Though usually the speed of the syncing process is bandwidth-bound so encrypting twice is not slowing things too much.
- have a command line option to force encryption or no encryption
- other ?
3  General Category / Feature Requests / Using S3Sync through a proxy server on: December 07, 2007, 05:00:34 AM
Unless I am mistaken, s3sync will not work if you have to go through a proxy server to reach the net.

A quick modification solved that for me (although it seems to break the --progress option  Grin).

In AWSAuthConnection::initialize

replace
Code:
        @http = Net::HTTP.new(server, port)

with
Code:
        if $PROXY_HOST
          @http = Net::HTTP::Proxy($PROXY_HOST, $PROXY_PORT, $PROXY_USER, $PROXY_PASS).new(server, port)       
        else
          @http = Net::HTTP.new(server, port)
        end

And set the constants $PROXY_HOST, $PROXY_PORT, $PROXY_USER, $PROXY_PASS in your config.yml file.

Besides the --progress having no effect apparently, the rest seems to work fine.
Pages: [1]
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2013, Simple Machines Valid XHTML 1.0! Valid CSS!