Local Directory Structure

[timdau]

I am trying to use s3sync to backup the AWS files locally.

My AWS bucket has a directory structure and I want to mirror that locally.  But s3sync fails if the local directory doesn't already exist.

If I create the directory, the downloads the files and then complains about the next directory.

[maelcum]

What version are you using? Have you synced TO aws successfully, yet?

Whats the error exactly? Can you copy/paste the error line(s)?

Whats the command line you are using? It should be something along the lines of

Code:

s3sync.rb  -v -r -s  BUCKET:SOURCE  TARGET

assuming your variables are all set (which you have, otherwise it wouldn't start) and you are using ssl-encrypted traffic (-s).

[ferrix]

I know it's confusing bit this isn't a bug, just a design/usage limitation.  There are already lots of posts about this if you want to learn more.  Try Search.

[timdau]

maelcum: I don't want to TO s3.  I am backing up the files FROM s3.  In fact, my application would be totally broken if I ever sync'ed to S3.

I am running version 1.1.4

Here is the error:
# ./s3sync.rb -r -v --progress BUCKET: /usr/local/backup/s3_aws/BUCKET
Create node /100/test.jpg
./s3sync.rb:614:in `initialize': No such file or directory - /usr/local/backup/s3_aws/BUCKET/100/test.jpg.s3syncTemp (Errno::ENOENT)
        from ./s3sync.rb:614:in `open'
        from ./s3sync.rb:614:in `updateFrom'
        from ./s3sync.rb:378:in `main'
        from ./s3sync.rb:708

[timdau]

ferrix:The result of your "design decision" is the tool is only doing half of the job.  I did search, and I did see your earlier post.  No one has clearly articulated the problem earlier, so I am trying.  This tool _should_ be able to copy files from S3 to local.  But it can't.  As it is, it is only useful for uploading.  You might as well remove the function to download.  That way you aren't setting the wrong expectation.

[ferrix]

Actually the download works fine for restoring stuff uploaded by s3sync in the first place.  I understand this isn't the use case that you want, but still it is misleading to imply the download is not useful at all.

I am leaving this thread in the bugs section to remind me to add the feature to create local folders on the fly even when the folder nodes don't exist on the S3 side.  I *do* want to help but the issue is not simple to handle correctly.

As for your other thread about why it keeps trying to get more copies even after transferring once.. that seems like a bug and I will investigate it.

[timdau]

ferrix:  I appreciate your reply.  The folder nodes DO exist on the S3 side.  Just not on the local side.  I want to duplicate the layout I have on S3 locally.

[maelcum]

timdau:

Code:

maelcum: I don't want to TO s3.  I am backing up the files FROM s3.  In fact, my application would be totally broken if I ever sync'ed to S3.

I was trying to help. Checking if things were uploaded, because then they do download fine at my place here.
But I guess you weren't appreciating help. Sorry I tried. Waste of time.
 

[ferrix]

maelcum my guess is the frustration must be not with you but with timdau's overall S3 tools experience.  I can relate: when I wanted some way to sync local and S3 content I was very frustrated.  I ended up making my own! 

And now that s3sync exists and is popular, there is a whole new target for people to express frustrations at

[ferrix]

ferrix:  I appreciate your reply.  The folder nodes DO exist on the S3 side.  Just not on the local side.  I want to duplicate the layout I have on S3 locally.

If you didn't use s3sync to store it (which we already know you didn't) then no there aren't any *s3sync* folder nodes.  In S3 there's no actual concept of folders, just nodes.   So, every tool uses its own proprietary way of storing this info (my scheme being the best naturally) and in general the methods are not compatible.

Don't worry though because by popular demand I am adding --make-dirs to the next version.

[timdau]

I see that I didn't do enough research on this issue.  Sorry for being a jerk.

[ferrix]

Think all the issues in here are addressed now so, moving.