Well, I found out more information and it is not good news. It turns out that a small portion of my data is getting corrupt. Boy am I glad I decided to investigate further. I STRONGLY URGE EVERYONE TO VERIFY THEIR TRANSFERS ARE NOT RANDOMLY RE-UPDATING BY USING THE -V FLAG.
What I did was I starting double checking all of the md5sums and there were indeed some that were different. I then starting doing binary differing and the corrupt files were only a few bytes different. The file sizes were all equal but in all cases there were 1-4 extra bytes zeroed out at the end of the file. Also, in about half of the cases there were also 2-10 consecutive bytes in the middle of the file that were just wrong.
I tried removing the the --ssl option to see if that had something to do with it but it made no different. I also tried adding in the the '--content-md5' patch from the '
http://s3sync.net/forum/index.php?topic=206.0' topic. Using that new command line option made no difference either.
My only conclusion is since I'm doing md5 checks that somehow the file is corrupted prior to the actual upload. Perhaps this is caused by a bad kernel compile or bad memory, but I'm surprise I'm not having had more serious issues then. It is also possible that amazon made a change that s3sync is not completely compatible. I noticed that there hasn't been any new development since June 2008--I would think that the content-md5 patch would be included by now.
Well, I'm going to start trying some alternative s3 backup methods just to see if I have similar problems. I do really enjoy s3sync as does exactly what I want but I can't deal with possible corruptions occurring and I don't think I can troubleshoot this much further.