February 02, 2014, 01:31:15 PM *
Welcome, Guest. Please login or register.

Login with username, password and session length
   Home   Help Search Login Register  
Pages: [1]
Author Topic: elastic mapreduce, Panic Transmit and S3sync.  (Read 5344 times)
Posts: 1

View Profile
« on: September 23, 2009, 06:10:15 PM »

I've been trying to use S3sync to get results back from running jobs on elastic map-reduce. I set up my bucket just so:

wd-emr:results  using Panic Transmit.

Now I get the reported error (below). DIgging around, I found this discussion: http://s3sync.net/forum/index.php?topic=190.0 

However, this is *EXACTLY* what elastic mapreduce needs to function -- its asked for an output S3bucket and key wd-emr:results
but this is exactly the key that causes problem for s3sync. I look forward to your comments. The obvious (but not palatable) solution is to use independent buckets, like wd-emr-results: wd-emr-logs and so on. Not happy with this idea though, seems to needlessly multiply buckets.

s3Prefix cache
localPrefix /Users/wdavies/emr/tmp/cache
s3TreeRecurse wd-emr cache
Creating new connection
Trying command list_bucket wd-emr max-keys 200 prefix cache delimiter / with 100 retries left
Response code: 200
prefix found: /
s3TreeRecurse wd-emr cache /
Trying command list_bucket wd-emr max-keys 200 prefix cache/ delimiter / with 100 retries left
Response code: 200
S3 item cache/
s3 node object init. Name: Path:cache Size:10 Tag:7fcfa567f82df949ccc30da8c6454e72 Date:Tue Sep 22 19:02:05 UTC 2009
local node object init. Name: Path:/Users/wdavies/emr/tmp/cache/ Size:38 Tag:d66759af42f282e1ba19144df2d405d0 Date:Wed Sep 23 22:48:11 UTC 2009
Update node
Trying command get_stream wd-emr cache  #<File:0x61b28c> with 100 retries left
Response code: 404
S3 command failed:
get_stream cache  #<File:0x61b28c>
With result 404 Not Found
s3sync/s3sync.rb:645:in `unlink': Operation not permitted - /Users/wdavies/emr/tmp/cache/ (Errno::EPERM)
   from s3sync/s3sync.rb:645:in `updateFrom'
   from s3sync/s3sync.rb:393:in `main'
   from s3sync/s3sync.rb:735
Sr. Member
Posts: 363

(I am greg13070 on AWS forum)

View Profile
« Reply #1 on: September 26, 2009, 11:12:43 AM »

I'm sorry that the s3sync code is so fragile with respect to delimiters.  I mean to rewrite the entire project, but have not had time.
Pages: [1]
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2013, Simple Machines Valid XHTML 1.0! Valid CSS!