S3Sync.net
February 02, 2014, 01:19:00 PM *
Welcome, Guest. Please login or register.

Login with username, password and session length
 
   Home   Help Search Login Register  
Pages: [1]
  Print  
Author Topic: S3Sync Creating Extraneous 38 Byte File  (Read 5507 times)
jeremylemaire
Newbie
*
Posts: 2


View Profile
« on: November 05, 2010, 02:09:35 PM »

I am using the following script to sync a local Linux ext3 directory with S3:

s3sync.rb -r --progress --ssl --delete /home/lucid/luciddb/archive mybucket:lucid-backup

For each directory in the structure I am seeing a 38 byte file with the same name as the directory name.  This is taking up a substantial amount of S3 space and more importantly is causing problems when trying to use EC2 MapReduce Hive scripts to create external tables.

  • Why are these extraneous files being uploaded?
  • Is there a way to prevent S3Sync from uploading these extraneous files?
Logged
jeremylemaire
Newbie
*
Posts: 2


View Profile
« Reply #1 on: November 08, 2010, 12:35:51 PM »

After reading this post http://s3sync.net/forum/index.php?topic=39.msg143#msg143 and tinkering a bit I have come to the conclusion that this 38 byte file is meta data used by s3sync to mimic the hierarchal directory structure being uploaded to S3.

Unless I missed it, I think this should be added to the README.txt in the s3sync install.


 
Logged
Admin
Administrator
Newbie
*****
Posts: 10


View Profile
« Reply #2 on: November 16, 2010, 06:55:30 PM »

To just upload items with no extra data needed for syncing, you could use the s3cmd.rb in an external script I guess.

It's not a good solution, but then again s3sync is really outdated too.  I have not had time to rewrite it yet, and no one else volunteers Smiley
Logged
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2013, Simple Machines Valid XHTML 1.0! Valid CSS!