S3Sync.net
February 02, 2014, 01:29:54 PM *
Welcome, Guest. Please login or register.

Login with username, password and session length
 
  Home Help Search Login Register  
  Show Posts
Pages: 1 ... 3 4 [5] 6 7 ... 25
61  General Category / Questions / Re: "list_bucket max-keys 200" error on: July 15, 2008, 01:55:42 PM
It's just part of the command that "failed".  500 errors are normal in S3, and means "try again shortly"
62  General Category / Questions / Re: Want a log, but not what --progress does on: July 08, 2008, 03:02:37 PM
-v
63  General Category / Announcements / Consider how this AWS error may affect your data on: June 23, 2008, 09:28:00 PM
http://developer.amazonwebservices.com/connect/thread.jspa?messageID=93408

Due to the fact that s3sync does not (ever, yet) employ the "Content-MD5" header, anyone using s3sync may have stored corrupted data to S3 during the interval discussed above.

Additionally, if you are in the habit of using "--no-md5", as I must do on some of my servers, such corruption will continue to escape notice on future syncs.  This is because there's no other way to check actual node contents vs local.

If possible, the optimal solution would be to run all your local->s3 syncs allowing for md5 comparison (which is the default).  Any corrupt nodes will be detected as changed, and re-synced.

64  General Category / Report Bugs / Should use Content-MD5 header on PUT on: June 23, 2008, 02:25:46 PM
Especially in light of http://developer.amazonwebservices.com/connect/thread.jspa?threadID=22709
65  General Category / Questions / Re: to ssl or not? on: June 20, 2008, 07:37:51 PM
but does that then mean my s3 id and access key will be sent in plain text?

No S3 is designed to always protect your secret key.  SSL is only for securing the contents of the nodes in-flight from eavesdropping.
66  General Category / Announcements / Version 1.2.6 is released on: June 16, 2008, 02:34:51 PM
Version 1.2.6
Catch connect errors and retry.
67  General Category / Report Bugs / Re: unrescued exception on: June 16, 2008, 02:34:30 PM
Good catch. 

Version 1.2.6
Catch connect errors and retry.
68  General Category / General Discussion / Re: s3sync gem on RubyForge on: May 24, 2008, 09:11:20 PM
Thank you, cloveless.  After your hard work I hope one day you attain enough success to get cloves.  Unless you are cloveless because you don't like cloves, of course; in which case carry on.
69  General Category / Questions / Re: s3cmd.rb to create subfolders? on: May 22, 2008, 07:16:24 PM
The $folder thing is used by some other tool but not mine.  To put a node that looks like a folder to s3sync you'd have to make a file with the exact content:
Quote
{E40327BF-517A-46e8-A6C3-AF51BC263F59}

That hashes to the MD5 that s3sync recognizes as a folder.  Don't try to use my stuff to make folders that other tools will play nice with; there's no standard and no one does it the same way.  (my way is best Wink

But what I don't get is, why do you care?  There's no reason to invent empty folders.  Just store whatever/path/you/want/keyname.
70  General Category / Report Bugs / Re: Transfers randomly dying on: May 21, 2008, 06:01:36 PM
I don't get what is causing that error here.. But I'll catch it starting in the next ver.
71  General Category / General Discussion / Re: 403 Forbidden on: May 18, 2008, 08:14:51 AM
mybucket:/testingatempfolder is incorrect
use mybucket:testingatempfolder
72  General Category / Report Bugs / Re: Syncing down from S3 fails to create directories (w/ --sync-dirs) on: May 12, 2008, 02:51:42 AM
Aha!  How do you have an item named "Woodworking/" in the bucket!?  That's what it's choking on.  We use / as a delimeter, it should never occur at the end of a key.
73  General Category / Report Bugs / Re: Syncing down from S3 fails to create directories (w/ --sync-dirs) on: May 11, 2008, 01:30:16 PM
Unlink errors?  Man, I have no idea what's going on with your stuff.  Maybe if you get -d output.  But I dunno, I think you're just cursed.
74  General Category / Announcements / Version 1.2.5 is released on: May 11, 2008, 02:38:40 AM
Added option --no-md5

About MD5 hashes
----------------
s3sync's normal operation is to compare the file size and MD5 hash of each item
to decide whether it needs syncing.  On the S3 side, these hashes are stored and
returned to us as the "ETag" of each item when the bucket is listed, so it's
very easy.  On the local side, the MD5 must be calculated by pushing every byte
in the file through the MD5 algorithm.  This is CPU and IO intensive! 

Thus you can specify the option --no-md5. This will compare the upload time on
S3 to the "last modified" time on the local item, and not do md5 calculations
locally at all. This might cause more transfers than are absolutely necessary.
For example if the file is "touched" to a newer modified date, but its contents
didn't change. Conversely if a file's contents are modified but the date is not
updated, then the sync will pass over it.  Lastly, if your clock is very
different from the one on the S3 servers, then you may see unanticipated
behavior.
75  General Category / Questions / Re: s3cmd get use a lot or resource, 10 times more CPU the put on: May 11, 2008, 02:37:42 AM
Like I said, the http ruby lib is awful.  You might try to profile it if you want to see what's going on.  But my code does not do anything cpu intensive as far as I know.
Pages: 1 ... 3 4 [5] 6 7 ... 25
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2013, Simple Machines Valid XHTML 1.0! Valid CSS!