S3Sync.net
February 02, 2014, 01:19:55 PM *
Welcome, Guest. Please login or register.

Login with username, password and session length
 
   Home   Help Search Login Register  
Pages: [1]
  Print  
Author Topic: Local Directory Structure  (Read 10800 times)
timdau
Newbie
*
Posts: 8


View Profile
« on: November 19, 2007, 01:00:51 PM »

I am trying to use s3sync to backup the AWS files locally.

My AWS bucket has a directory structure and I want to mirror that locally.  But s3sync fails if the local directory doesn't already exist.

If I create the directory, the downloads the files and then complains about the next directory.
Logged
maelcum
Newbie
*
Posts: 43



View Profile
« Reply #1 on: November 19, 2007, 01:29:55 PM »

What version are you using? Have you synced TO aws successfully, yet?

Whats the error exactly? Can you copy/paste the error line(s)?

Whats the command line you are using? It should be something along the lines of
Code:
s3sync.rb  -v -r -s  BUCKET:SOURCE  TARGET
assuming your variables are all set (which you have, otherwise it wouldn't start) and you are using ssl-encrypted traffic (-s).

Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #2 on: November 19, 2007, 08:33:22 PM »

I know it's confusing bit this isn't a bug, just a design/usage limitation.  There are already lots of posts about this if you want to learn more.  Try Search.
Logged
timdau
Newbie
*
Posts: 8


View Profile
« Reply #3 on: November 20, 2007, 09:57:23 AM »

maelcum: I don't want to TO s3.  I am backing up the files FROM s3.  In fact, my application would be totally broken if I ever sync'ed to S3.

I am running version 1.1.4

Here is the error:
# ./s3sync.rb -r -v --progress BUCKET: /usr/local/backup/s3_aws/BUCKET
Create node /100/test.jpg
./s3sync.rb:614:in `initialize': No such file or directory - /usr/local/backup/s3_aws/BUCKET/100/test.jpg.s3syncTemp (Errno::ENOENT)
        from ./s3sync.rb:614:in `open'
        from ./s3sync.rb:614:in `updateFrom'
        from ./s3sync.rb:378:in `main'
        from ./s3sync.rb:708


Logged
timdau
Newbie
*
Posts: 8


View Profile
« Reply #4 on: November 20, 2007, 09:57:54 AM »

ferrix:The result of your "design decision" is the tool is only doing half of the job.  I did search, and I did see your earlier post.  No one has clearly articulated the problem earlier, so I am trying.  This tool _should_ be able to copy files from S3 to local.  But it can't.  As it is, it is only useful for uploading.  You might as well remove the function to download.  That way you aren't setting the wrong expectation.
Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #5 on: November 20, 2007, 02:52:43 PM »

Actually the download works fine for restoring stuff uploaded by s3sync in the first place.  I understand this isn't the use case that you want, but still it is misleading to imply the download is not useful at all.

I am leaving this thread in the bugs section to remind me to add the feature to create local folders on the fly even when the folder nodes don't exist on the S3 side.  I *do* want to help but the issue is not simple to handle correctly.

As for your other thread about why it keeps trying to get more copies even after transferring once.. that seems like a bug and I will investigate it.
Logged
timdau
Newbie
*
Posts: 8


View Profile
« Reply #6 on: November 20, 2007, 03:24:27 PM »

ferrix:  I appreciate your reply.  The folder nodes DO exist on the S3 side.  Just not on the local side.  I want to duplicate the layout I have on S3 locally.
Logged
maelcum
Newbie
*
Posts: 43



View Profile
« Reply #7 on: November 20, 2007, 03:29:40 PM »

timdau:
Code:
maelcum: I don't want to TO s3.  I am backing up the files FROM s3.  In fact, my application would be totally broken if I ever sync'ed to S3.
I was trying to help. Checking if things were uploaded, because then they do download fine at my place here.
But I guess you weren't appreciating help. Sorry I tried. Waste of time.
 Angry
Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #8 on: November 20, 2007, 05:53:32 PM »

maelcum my guess is the frustration must be not with you but with timdau's overall S3 tools experience.  I can relate: when I wanted some way to sync local and S3 content I was very frustrated.  I ended up making my own! 

And now that s3sync exists and is popular, there is a whole new target for people to express frustrations at Smiley
Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #9 on: November 20, 2007, 06:49:32 PM »

ferrix:  I appreciate your reply.  The folder nodes DO exist on the S3 side.  Just not on the local side.  I want to duplicate the layout I have on S3 locally.

If you didn't use s3sync to store it (which we already know you didn't) then no there aren't any *s3sync* folder nodes.  In S3 there's no actual concept of folders, just nodes.   So, every tool uses its own proprietary way of storing this info (my scheme being the best naturally) and in general the methods are not compatible.

Don't worry though because by popular demand I am adding --make-dirs to the next version.
Logged
timdau
Newbie
*
Posts: 8


View Profile
« Reply #10 on: November 20, 2007, 09:32:24 PM »

I see that I didn't do enough research on this issue.  Sorry for being a jerk.
Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #11 on: November 21, 2007, 12:13:58 AM »

Think all the issues in here are addressed now so, moving.
Logged
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2013, Simple Machines Valid XHTML 1.0! Valid CSS!