S3Sync.net

I am using the following script to sync a local Linux ext3 directory with S3:

s3sync.rb -r --progress --ssl --delete /home/lucid/luciddb/archive mybucket:lucid-backup

For each directory in the structure I am seeing a 38 byte file with the same name as the directory name.  This is taking up a substantial amount of S3 space and more importantly is causing problems when trying to use EC2 MapReduce Hive scripts to create external tables.

• Why are these extraneous files being uploaded?
• Is there a way to prevent S3Sync from uploading these extraneous files?

After reading this post http://s3sync.net/forum/index.php?topic=39.msg143#msg143 (http://s3sync.net/forum/index.php?topic=39.msg143#msg143) and tinkering a bit I have come to the conclusion that this 38 byte file is meta data used by s3sync to mimic the hierarchal directory structure being uploaded to S3.

Unless I missed it, I think this should be added to the README.txt in the s3sync install.


 

To just upload items with no extra data needed for syncing, you could use the s3cmd.rb in an external script I guess.

It's not a good solution, but then again s3sync is really outdated too.  I have not had time to rewrite it yet, and no one else volunteers :)