S3Sync.net
February 02, 2014, 01:19:46 PM *
Welcome, Guest. Please login or register.

Login with username, password and session length
 
   Home   Help Search Login Register  
Pages: [1]
  Print  
Author Topic: Syncing down from S3 fails to create directories (w/ --sync-dirs)  (Read 9103 times)
dbingham
Newbie
*
Posts: 4


View Profile
« on: May 09, 2008, 11:14:22 AM »

I've seen the various other posts related to problems syncing down from S3 when the UP was done with another program.  My understanding was that "--make-dirs" was implemented to deal with that.  However, I'm running the command with that option and it still isn't working for me:

Code:
dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r DLBProjects:Woodworking /Users/dbingham/temp/
./s3sync.rb:627:in `initialize': No such file or directory - /Users/dbingham/temp/Woodworking/.s3syncTemp (Errno::ENOENT)
from ./s3sync.rb:627:in `open'
from ./s3sync.rb:627:in `updateFrom'
from ./s3sync.rb:392:in `main'
from ./s3sync.rb:724

Thanks.
Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #1 on: May 10, 2008, 08:14:49 PM »

It might not be perfect.  Try creating the top level directory it's stuck on.
Logged
dbingham
Newbie
*
Posts: 4


View Profile
« Reply #2 on: May 11, 2008, 10:24:18 AM »

I tried two ways to do what you suggested:

Attempt 1:
Code:
dbingham:~/Scripts/s3sync dbingham$ mkdir ~/temp/Woodworking
dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r DLBProjects:Woodworking /Users/dbingham/temp/
S3 command failed:
get_stream Woodworking  #<File:0x61b6c4>
With result 404 Not Found
./s3sync.rb:634:in `unlink': Operation not permitted - /Users/dbingham/temp/Woodworking/ (Errno::EPERM)
from ./s3sync.rb:634:in `updateFrom'
from ./s3sync.rb:392:in `main'
from ./s3sync.rb:724

Attempt 2:
Code:
dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r DLBProjects:Woodworking/ /Users/dbingham/temp/Woodworking/
S3 command failed:
get_stream Woodworking  #<File:0x61bd40>
With result 404 Not Found
./s3sync.rb:634:in `unlink': Operation not permitted - /Users/dbingham/temp/Woodworking/ (Errno::EPERM)
from ./s3sync.rb:634:in `updateFrom'
from ./s3sync.rb:392:in `main'
from ./s3sync.rb:724

And just to make sure I'm not crazy:
Code:
dbingham:~/Scripts/s3sync dbingham$ ./s3cmd.rb list DLBProjects
--------------------
Woodworking/
Woodworking/BFCX_151.dmg
Woodworking/Boardfoot CalculatorX 1.5
Woodworking/Cabinets/
Woodworking/Cabinets/BathroomCabinets.skp
Woodworking/Cabinets/BathroomCabinets2.skp
Woodworking/Cabinets/BathroomCabinets2~.skp
Woodworking/Cabinets/BathroomCabinets~.skp
Woodworking/Cutting Board.pdf
Woodworking/sharpening.pdf
Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #3 on: May 11, 2008, 01:30:16 PM »

Unlink errors?  Man, I have no idea what's going on with your stuff.  Maybe if you get -d output.  But I dunno, I think you're just cursed.
Logged
dbingham
Newbie
*
Posts: 4


View Profile
« Reply #4 on: May 11, 2008, 10:14:09 PM »

The three trials with "-d":

1. Without manually creating target directory:
Code:
dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r -d DLBProjects:Woodworking /Users/dbingham/temp/
s3Prefix Woodworking
localPrefix /Users/dbingham/temp/Woodworking
s3TreeRecurse DLBProjects Woodworking
Creating new connection
Trying command list_bucket DLBProjects max-keys 200 prefix Woodworking delimiter / with 100 retries left
Response code: 200
prefix found: /
s3TreeRecurse DLBProjects Woodworking /
Trying command list_bucket DLBProjects max-keys 200 prefix Woodworking/ delimiter / with 100 retries left
Response code: 200
S3 item Woodworking/
s3 node object init. Name: Path:Woodworking Size:0 Tag:d41d8cd98f00b204e9800998ecf8427e
local node object init. Name: Path:/Users/dbingham/temp/Woodworking/ Size: Tag:
source:
dest:
Update node
./s3sync.rb:627:in `initialize': No such file or directory - /Users/dbingham/temp/Woodworking/.s3syncTemp (Errno::ENOENT)
from ./s3sync.rb:627:in `open'
from ./s3sync.rb:627:in `updateFrom'
from ./s3sync.rb:392:in `main'
from ./s3sync.rb:724

2. Manually create target directory but use same command:
Code:
dbingham:~/Scripts/s3sync dbingham$ mkdir ~/temp/Woodworking
dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r -d DLBProjects:Woodworking /Users/dbingham/temp/
s3Prefix Woodworking
localPrefix /Users/dbingham/temp/Woodworking
s3TreeRecurse DLBProjects Woodworking
Creating new connection
Trying command list_bucket DLBProjects max-keys 200 prefix Woodworking delimiter / with 100 retries left
Response code: 200
prefix found: /
s3TreeRecurse DLBProjects Woodworking /
Trying command list_bucket DLBProjects max-keys 200 prefix Woodworking/ delimiter / with 100 retries left
Response code: 200
S3 item Woodworking/
s3 node object init. Name: Path:Woodworking Size:0 Tag:d41d8cd98f00b204e9800998ecf8427e
local node object init. Name: Path:/Users/dbingham/temp/Woodworking/ Size:38 Tag:d66759af42f282e1ba19144df2d405d0
source:
dest:
Update node
Trying command get_stream DLBProjects Woodworking  #<File:0x61b674> with 100 retries left
Response code: 404
S3 command failed:
get_stream Woodworking  #<File:0x61b674>
With result 404 Not Found
#<Net::ReadAdapter:0x61399c>
./s3sync.rb:634:in `unlink': Operation not permitted - /Users/dbingham/temp/Woodworking/ (Errno::EPERM)
from ./s3sync.rb:634:in `updateFrom'
from ./s3sync.rb:392:in `main'
from ./s3sync.rb:724

3. Manually create directory and add "/" at the end of the from to make it only copy content:
Code:
dbingham:~/Scripts/s3sync dbingham$ mkdir ~/temp/Woodworking
dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r -d DLBProjects:Woodworking/ /Users/dbingham/temp/Woodworking/
s3Prefix Woodworking/
localPrefix /Users/dbingham/temp/Woodworking
s3TreeRecurse DLBProjects Woodworking/
Creating new connection
Trying command list_bucket DLBProjects max-keys 200 prefix Woodworking/ delimiter / with 100 retries left
Response code: 200
S3 item Woodworking/
s3 node object init. Name: Path:Woodworking Size:0 Tag:d41d8cd98f00b204e9800998ecf8427e
local node object init. Name: Path:/Users/dbingham/temp/Woodworking/ Size:38 Tag:d66759af42f282e1ba19144df2d405d0
source:
dest:
Update node
Trying command get_stream DLBProjects Woodworking  #<File:0x61bd04> with 100 retries left
Response code: 404
S3 command failed:
get_stream Woodworking  #<File:0x61bd04>
With result 404 Not Found
#<Net::ReadAdapter:0x614f2c>
./s3sync.rb:634:in `unlink': Operation not permitted - /Users/dbingham/temp/Woodworking/ (Errno::EPERM)
from ./s3sync.rb:634:in `updateFrom'
from ./s3sync.rb:392:in `main'
from ./s3sync.rb:724

The first trial seems the most promising.  Kinda looks like it is trying to use the local "Woodworking" directory without actually creating it.  However, if #2 and #3 are any indication of what would happen in the next step, it wouldn't much matter anyway.

Seems to me like its trying to get a file named "#", but looking at my file listing above, there obviously is no such thing.  Not sure if I'm really interpreting that correctly, though.

Anyway, thanks for your help.
Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #5 on: May 12, 2008, 02:51:42 AM »

Aha!  How do you have an item named "Woodworking/" in the bucket!?  That's what it's choking on.  We use / as a delimeter, it should never occur at the end of a key.
Logged
dbingham
Newbie
*
Posts: 4


View Profile
« Reply #6 on: May 12, 2008, 11:48:36 AM »

Indeed you are correct.  Never really noticed as most clients just seem to ignore that.

Confirmed it with s3browse.com which seems to interpret the "Woodworking/" as a directory with a zero byte file in it with no name.

*sigh* the lack of a formal directory structure in S3 is a PITA.

Sorry for the false bug, but thanks for the help figuring it out.
Logged
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2013, Simple Machines Valid XHTML 1.0! Valid CSS!