Syncing down from S3 fails to create directories (w/ --sync-dirs)

[dbingham]

I've seen the various other posts related to problems syncing down from S3 when the UP was done with another program.  My understanding was that "--make-dirs" was implemented to deal with that.  However, I'm running the command with that option and it still isn't working for me:

Code:

dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r DLBProjects:Woodworking /Users/dbingham/temp/
./s3sync.rb:627:in `initialize': No such file or directory - /Users/dbingham/temp/Woodworking/.s3syncTemp (Errno::ENOENT)
from ./s3sync.rb:627:in `open'
from ./s3sync.rb:627:in `updateFrom'
from ./s3sync.rb:392:in `main'
from ./s3sync.rb:724

Thanks.

[ferrix]

It might not be perfect.  Try creating the top level directory it's stuck on.

[dbingham]

I tried two ways to do what you suggested:

Attempt 1:

Code:

dbingham:~/Scripts/s3sync dbingham$ mkdir ~/temp/Woodworking
dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r DLBProjects:Woodworking /Users/dbingham/temp/
S3 command failed:
get_stream Woodworking  #<File:0x61b6c4>
With result 404 Not Found
./s3sync.rb:634:in `unlink': Operation not permitted - /Users/dbingham/temp/Woodworking/ (Errno::EPERM)
from ./s3sync.rb:634:in `updateFrom'
from ./s3sync.rb:392:in `main'
from ./s3sync.rb:724

Attempt 2:

Code:

dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r DLBProjects:Woodworking/ /Users/dbingham/temp/Woodworking/
S3 command failed:
get_stream Woodworking  #<File:0x61bd40>
With result 404 Not Found
./s3sync.rb:634:in `unlink': Operation not permitted - /Users/dbingham/temp/Woodworking/ (Errno::EPERM)
from ./s3sync.rb:634:in `updateFrom'
from ./s3sync.rb:392:in `main'
from ./s3sync.rb:724

And just to make sure I'm not crazy:

Code:

dbingham:~/Scripts/s3sync dbingham$ ./s3cmd.rb list DLBProjects
--------------------
Woodworking/
Woodworking/BFCX_151.dmg
Woodworking/Boardfoot CalculatorX 1.5
Woodworking/Cabinets/
Woodworking/Cabinets/BathroomCabinets.skp
Woodworking/Cabinets/BathroomCabinets2.skp
Woodworking/Cabinets/BathroomCabinets2~.skp
Woodworking/Cabinets/BathroomCabinets~.skp
Woodworking/Cutting Board.pdf
Woodworking/sharpening.pdf

[ferrix]

Unlink errors?  Man, I have no idea what's going on with your stuff.  Maybe if you get -d output.  But I dunno, I think you're just cursed.

[dbingham]

The three trials with "-d":

1. Without manually creating target directory:

Code:

dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r -d DLBProjects:Woodworking /Users/dbingham/temp/
s3Prefix Woodworking
localPrefix /Users/dbingham/temp/Woodworking
s3TreeRecurse DLBProjects Woodworking
Creating new connection
Trying command list_bucket DLBProjects max-keys 200 prefix Woodworking delimiter / with 100 retries left
Response code: 200
prefix found: /
s3TreeRecurse DLBProjects Woodworking /
Trying command list_bucket DLBProjects max-keys 200 prefix Woodworking/ delimiter / with 100 retries left
Response code: 200
S3 item Woodworking/
s3 node object init. Name: Path:Woodworking Size:0 Tag:d41d8cd98f00b204e9800998ecf8427e
local node object init. Name: Path:/Users/dbingham/temp/Woodworking/ Size: Tag:
source:
dest:
Update node
./s3sync.rb:627:in `initialize': No such file or directory - /Users/dbingham/temp/Woodworking/.s3syncTemp (Errno::ENOENT)
from ./s3sync.rb:627:in `open'
from ./s3sync.rb:627:in `updateFrom'
from ./s3sync.rb:392:in `main'
from ./s3sync.rb:724

2. Manually create target directory but use same command:

Code:

dbingham:~/Scripts/s3sync dbingham$ mkdir ~/temp/Woodworking
dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r -d DLBProjects:Woodworking /Users/dbingham/temp/
s3Prefix Woodworking
localPrefix /Users/dbingham/temp/Woodworking
s3TreeRecurse DLBProjects Woodworking
Creating new connection
Trying command list_bucket DLBProjects max-keys 200 prefix Woodworking delimiter / with 100 retries left
Response code: 200
prefix found: /
s3TreeRecurse DLBProjects Woodworking /
Trying command list_bucket DLBProjects max-keys 200 prefix Woodworking/ delimiter / with 100 retries left
Response code: 200
S3 item Woodworking/
s3 node object init. Name: Path:Woodworking Size:0 Tag:d41d8cd98f00b204e9800998ecf8427e
local node object init. Name: Path:/Users/dbingham/temp/Woodworking/ Size:38 Tag:d66759af42f282e1ba19144df2d405d0
source:
dest:
Update node
Trying command get_stream DLBProjects Woodworking  #<File:0x61b674> with 100 retries left
Response code: 404
S3 command failed:
get_stream Woodworking  #<File:0x61b674>
With result 404 Not Found
#<Net::ReadAdapter:0x61399c>
./s3sync.rb:634:in `unlink': Operation not permitted - /Users/dbingham/temp/Woodworking/ (Errno::EPERM)
from ./s3sync.rb:634:in `updateFrom'
from ./s3sync.rb:392:in `main'
from ./s3sync.rb:724

3. Manually create directory and add "/" at the end of the from to make it only copy content:

Code:

dbingham:~/Scripts/s3sync dbingham$ mkdir ~/temp/Woodworking
dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r -d DLBProjects:Woodworking/ /Users/dbingham/temp/Woodworking/
s3Prefix Woodworking/
localPrefix /Users/dbingham/temp/Woodworking
s3TreeRecurse DLBProjects Woodworking/
Creating new connection
Trying command list_bucket DLBProjects max-keys 200 prefix Woodworking/ delimiter / with 100 retries left
Response code: 200
S3 item Woodworking/
s3 node object init. Name: Path:Woodworking Size:0 Tag:d41d8cd98f00b204e9800998ecf8427e
local node object init. Name: Path:/Users/dbingham/temp/Woodworking/ Size:38 Tag:d66759af42f282e1ba19144df2d405d0
source:
dest:
Update node
Trying command get_stream DLBProjects Woodworking  #<File:0x61bd04> with 100 retries left
Response code: 404
S3 command failed:
get_stream Woodworking  #<File:0x61bd04>
With result 404 Not Found
#<Net::ReadAdapter:0x614f2c>
./s3sync.rb:634:in `unlink': Operation not permitted - /Users/dbingham/temp/Woodworking/ (Errno::EPERM)
from ./s3sync.rb:634:in `updateFrom'
from ./s3sync.rb:392:in `main'
from ./s3sync.rb:724

The first trial seems the most promising.  Kinda looks like it is trying to use the local "Woodworking" directory without actually creating it.  However, if #2 and #3 are any indication of what would happen in the next step, it wouldn't much matter anyway.

Seems to me like its trying to get a file named "#", but looking at my file listing above, there obviously is no such thing.  Not sure if I'm really interpreting that correctly, though.

Anyway, thanks for your help.

[ferrix]

Aha!  How do you have an item named "Woodworking/" in the bucket!?  That's what it's choking on.  We use / as a delimeter, it should never occur at the end of a key.

[dbingham]

Indeed you are correct.  Never really noticed as most clients just seem to ignore that.

Confirmed it with s3browse.com which seems to interpret the "Woodworking/" as a directory with a zero byte file in it with no name.

*sigh* the lack of a formal directory structure in S3 is a PITA.

Sorry for the false bug, but thanks for the help figuring it out.