Title: Syncing down from S3 fails to create directories (w/ --sync-dirs) Post by: dbingham on May 09, 2008, 11:14:22 AM I've seen the various other posts related to problems syncing down from S3 when the UP was done with another program. My understanding was that "--make-dirs" was implemented to deal with that. However, I'm running the command with that option and it still isn't working for me:
Code: dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r DLBProjects:Woodworking /Users/dbingham/temp/ ./s3sync.rb:627:in `initialize': No such file or directory - /Users/dbingham/temp/Woodworking/.s3syncTemp (Errno::ENOENT) from ./s3sync.rb:627:in `open' from ./s3sync.rb:627:in `updateFrom' from ./s3sync.rb:392:in `main' from ./s3sync.rb:724 Thanks. Title: Re: Syncing down from S3 fails to create directories (w/ --sync-dirs) Post by: ferrix on May 10, 2008, 08:14:49 PM It might not be perfect. Try creating the top level directory it's stuck on.
Title: Re: Syncing down from S3 fails to create directories (w/ --sync-dirs) Post by: dbingham on May 11, 2008, 10:24:18 AM I tried two ways to do what you suggested:
Attempt 1: Code: dbingham:~/Scripts/s3sync dbingham$ mkdir ~/temp/Woodworking dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r DLBProjects:Woodworking /Users/dbingham/temp/ S3 command failed: get_stream Woodworking #<File:0x61b6c4> With result 404 Not Found ./s3sync.rb:634:in `unlink': Operation not permitted - /Users/dbingham/temp/Woodworking/ (Errno::EPERM) from ./s3sync.rb:634:in `updateFrom' from ./s3sync.rb:392:in `main' from ./s3sync.rb:724 Attempt 2: Code: dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r DLBProjects:Woodworking/ /Users/dbingham/temp/Woodworking/ S3 command failed: get_stream Woodworking #<File:0x61bd40> With result 404 Not Found ./s3sync.rb:634:in `unlink': Operation not permitted - /Users/dbingham/temp/Woodworking/ (Errno::EPERM) from ./s3sync.rb:634:in `updateFrom' from ./s3sync.rb:392:in `main' from ./s3sync.rb:724 And just to make sure I'm not crazy: Code: dbingham:~/Scripts/s3sync dbingham$ ./s3cmd.rb list DLBProjects -------------------- Woodworking/ Woodworking/BFCX_151.dmg Woodworking/Boardfoot CalculatorX 1.5 Woodworking/Cabinets/ Woodworking/Cabinets/BathroomCabinets.skp Woodworking/Cabinets/BathroomCabinets2.skp Woodworking/Cabinets/BathroomCabinets2~.skp Woodworking/Cabinets/BathroomCabinets~.skp Woodworking/Cutting Board.pdf Woodworking/sharpening.pdf Title: Re: Syncing down from S3 fails to create directories (w/ --sync-dirs) Post by: ferrix on May 11, 2008, 01:30:16 PM Unlink errors? Man, I have no idea what's going on with your stuff. Maybe if you get -d output. But I dunno, I think you're just cursed.
Title: Re: Syncing down from S3 fails to create directories (w/ --sync-dirs) Post by: dbingham on May 11, 2008, 10:14:09 PM The three trials with "-d":
1. Without manually creating target directory: Code: dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r -d DLBProjects:Woodworking /Users/dbingham/temp/ s3Prefix Woodworking localPrefix /Users/dbingham/temp/Woodworking s3TreeRecurse DLBProjects Woodworking Creating new connection Trying command list_bucket DLBProjects max-keys 200 prefix Woodworking delimiter / with 100 retries left Response code: 200 prefix found: / s3TreeRecurse DLBProjects Woodworking / Trying command list_bucket DLBProjects max-keys 200 prefix Woodworking/ delimiter / with 100 retries left Response code: 200 S3 item Woodworking/ s3 node object init. Name: Path:Woodworking Size:0 Tag:d41d8cd98f00b204e9800998ecf8427e local node object init. Name: Path:/Users/dbingham/temp/Woodworking/ Size: Tag: source: dest: Update node ./s3sync.rb:627:in `initialize': No such file or directory - /Users/dbingham/temp/Woodworking/.s3syncTemp (Errno::ENOENT) from ./s3sync.rb:627:in `open' from ./s3sync.rb:627:in `updateFrom' from ./s3sync.rb:392:in `main' from ./s3sync.rb:724 2. Manually create target directory but use same command: Code: dbingham:~/Scripts/s3sync dbingham$ mkdir ~/temp/Woodworking dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r -d DLBProjects:Woodworking /Users/dbingham/temp/ s3Prefix Woodworking localPrefix /Users/dbingham/temp/Woodworking s3TreeRecurse DLBProjects Woodworking Creating new connection Trying command list_bucket DLBProjects max-keys 200 prefix Woodworking delimiter / with 100 retries left Response code: 200 prefix found: / s3TreeRecurse DLBProjects Woodworking / Trying command list_bucket DLBProjects max-keys 200 prefix Woodworking/ delimiter / with 100 retries left Response code: 200 S3 item Woodworking/ s3 node object init. Name: Path:Woodworking Size:0 Tag:d41d8cd98f00b204e9800998ecf8427e local node object init. Name: Path:/Users/dbingham/temp/Woodworking/ Size:38 Tag:d66759af42f282e1ba19144df2d405d0 source: dest: Update node Trying command get_stream DLBProjects Woodworking #<File:0x61b674> with 100 retries left Response code: 404 S3 command failed: get_stream Woodworking #<File:0x61b674> With result 404 Not Found #<Net::ReadAdapter:0x61399c> ./s3sync.rb:634:in `unlink': Operation not permitted - /Users/dbingham/temp/Woodworking/ (Errno::EPERM) from ./s3sync.rb:634:in `updateFrom' from ./s3sync.rb:392:in `main' from ./s3sync.rb:724 3. Manually create directory and add "/" at the end of the from to make it only copy content: Code: dbingham:~/Scripts/s3sync dbingham$ mkdir ~/temp/Woodworking dbingham:~/Scripts/s3sync dbingham$ ./s3sync.rb --make-dirs -r -d DLBProjects:Woodworking/ /Users/dbingham/temp/Woodworking/ s3Prefix Woodworking/ localPrefix /Users/dbingham/temp/Woodworking s3TreeRecurse DLBProjects Woodworking/ Creating new connection Trying command list_bucket DLBProjects max-keys 200 prefix Woodworking/ delimiter / with 100 retries left Response code: 200 S3 item Woodworking/ s3 node object init. Name: Path:Woodworking Size:0 Tag:d41d8cd98f00b204e9800998ecf8427e local node object init. Name: Path:/Users/dbingham/temp/Woodworking/ Size:38 Tag:d66759af42f282e1ba19144df2d405d0 source: dest: Update node Trying command get_stream DLBProjects Woodworking #<File:0x61bd04> with 100 retries left Response code: 404 S3 command failed: get_stream Woodworking #<File:0x61bd04> With result 404 Not Found #<Net::ReadAdapter:0x614f2c> ./s3sync.rb:634:in `unlink': Operation not permitted - /Users/dbingham/temp/Woodworking/ (Errno::EPERM) from ./s3sync.rb:634:in `updateFrom' from ./s3sync.rb:392:in `main' from ./s3sync.rb:724 The first trial seems the most promising. Kinda looks like it is trying to use the local "Woodworking" directory without actually creating it. However, if #2 and #3 are any indication of what would happen in the next step, it wouldn't much matter anyway. Seems to me like its trying to get a file named "#", but looking at my file listing above, there obviously is no such thing. Not sure if I'm really interpreting that correctly, though. Anyway, thanks for your help. Title: Re: Syncing down from S3 fails to create directories (w/ --sync-dirs) Post by: ferrix on May 12, 2008, 02:51:42 AM Aha! How do you have an item named "Woodworking/" in the bucket!? That's what it's choking on. We use / as a delimeter, it should never occur at the end of a key.
Title: Re: Syncing down from S3 fails to create directories (w/ --sync-dirs) Post by: dbingham on May 12, 2008, 11:48:36 AM Indeed you are correct. Never really noticed as most clients just seem to ignore that.
Confirmed it with s3browse.com which seems to interpret the "Woodworking/" as a directory with a zero byte file in it with no name. *sigh* the lack of a formal directory structure in S3 is a PITA. Sorry for the false bug, but thanks for the help figuring it out. |