Show Posts
|
Pages: 1 ... 3 4 [5] 6 7 ... 25
|
63
|
General Category / Announcements / Consider how this AWS error may affect your data
|
on: June 23, 2008, 09:28:00 PM
|
http://developer.amazonwebservices.com/connect/thread.jspa?messageID=93408Due to the fact that s3sync does not (ever, yet) employ the "Content-MD5" header, anyone using s3sync may have stored corrupted data to S3 during the interval discussed above. Additionally, if you are in the habit of using "--no-md5", as I must do on some of my servers, such corruption will continue to escape notice on future syncs. This is because there's no other way to check actual node contents vs local. If possible, the optimal solution would be to run all your local->s3 syncs allowing for md5 comparison (which is the default). Any corrupt nodes will be detected as changed, and re-synced.
|
|
|
65
|
General Category / Questions / Re: to ssl or not?
|
on: June 20, 2008, 07:37:51 PM
|
but does that then mean my s3 id and access key will be sent in plain text?
No S3 is designed to always protect your secret key. SSL is only for securing the contents of the nodes in-flight from eavesdropping.
|
|
|
69
|
General Category / Questions / Re: s3cmd.rb to create subfolders?
|
on: May 22, 2008, 07:16:24 PM
|
The $folder thing is used by some other tool but not mine. To put a node that looks like a folder to s3sync you'd have to make a file with the exact content: {E40327BF-517A-46e8-A6C3-AF51BC263F59} That hashes to the MD5 that s3sync recognizes as a folder. Don't try to use my stuff to make folders that other tools will play nice with; there's no standard and no one does it the same way. (my way is best But what I don't get is, why do you care? There's no reason to invent empty folders. Just store whatever/path/you/want/keyname.
|
|
|
74
|
General Category / Announcements / Version 1.2.5 is released
|
on: May 11, 2008, 02:38:40 AM
|
Added option --no-md5
About MD5 hashes ---------------- s3sync's normal operation is to compare the file size and MD5 hash of each item to decide whether it needs syncing. On the S3 side, these hashes are stored and returned to us as the "ETag" of each item when the bucket is listed, so it's very easy. On the local side, the MD5 must be calculated by pushing every byte in the file through the MD5 algorithm. This is CPU and IO intensive!
Thus you can specify the option --no-md5. This will compare the upload time on S3 to the "last modified" time on the local item, and not do md5 calculations locally at all. This might cause more transfers than are absolutely necessary. For example if the file is "touched" to a newer modified date, but its contents didn't change. Conversely if a file's contents are modified but the date is not updated, then the sync will pass over it. Lastly, if your clock is very different from the one on the S3 servers, then you may see unanticipated behavior.
|
|
|
|