S3Sync.net
February 02, 2014, 01:28:03 PM *
Welcome, Guest. Please login or register.

Login with username, password and session length
 
   Home   Help Search Login Register  
Pages: [1] 2 3 4
  Print  
Author Topic: How to choose storage location in Amazon S3  (Read 32302 times)
maelcum
Newbie
*
Posts: 43



View Profile
« on: November 06, 2007, 09:18:18 AM »

Hi,

not a s3sync/s3cmd-question, but concerning data storage in Amazon S3...

Since there is a choice to store data in the US or Europe, I'm trying to find out how this could be done.
So far I came across a FAQ which says that this decision is made when creating a bucket. I haven't found out how I have to name my bucket, though, to make sure its in the US (or Europe - where it is more expensive btw).

Could somebody point me into the right direction?

Thanks very much.

Cheers

maelcum
Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #1 on: November 06, 2007, 02:32:56 PM »

I don't know the answer, but I just wanted to say that I read your post and am not ignoring you Cheesy Cheesy
Logged
PolarLava
Newbie
*
Posts: 2


View Profile WWW
« Reply #2 on: November 06, 2007, 03:22:54 PM »

"To support Europe we enhanced Amazon S3’s API to include a new parameter for locality. The name of the parameter is LocationConstraint, and if you use it then buckets are created in the location you specify. Existing applications will not be affected, and will create buckets in the United States, because that’s the “default” behavior. Pricing is slightly different in Europe; although not but much. Use the AWS Price calculator if you want to estimate what your costs will be."

This comes from the AWS Blog http://aws.typepad.com/aws/2007/11/notes-from-berl.html

I would assume this will require some change to S3Sync to support selecting whether the bucket is created in the US or Europe.

-klp
Logged
Clou
Newbie
*
Posts: 10


View Profile
« Reply #3 on: November 07, 2007, 05:16:40 AM »

I created a "european" bucket with S3Fox, then tried to fill it with s3sync.rb.
Filling failed with:



./s3sync.rb --verbose /var/archives/ testeu:test
S3 command failed:
list_bucket testeu max-keys 200 prefix test delimiter /
With result 301 Moved Permanently
S3 ERROR: #<Net::HTTPMovedPermanently:0xb7ad27a0>
./s3sync.rb:292:in `+': can't convert nil into Array (TypeError)
        from ./s3sync.rb:292:in `s3TreeRecurse'
        from ./s3sync.rb:348:in `main'
        from ./thread_generator.rb:79:in `call'
        from ./thread_generator.rb:79:in `initialize'
        from ./thread_generator.rb:76:in `new'
        from ./thread_generator.rb:76:in `initialize'
        from ./s3sync.rb:269:in `new'
        from ./s3sync.rb:269:in `main'
        from ./s3sync.rb:708
Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #4 on: November 07, 2007, 10:05:39 AM »

I don't have one of those buckets but I'm guessing that the server name might be different?  You can set that via the env variable AWS_S3_HOST (defaults to s3.amazonaws.com)
Logged
maelcum
Newbie
*
Posts: 43



View Profile
« Reply #5 on: November 07, 2007, 10:15:18 AM »


@Ferrix:
Quote
I don't know the answer, but I just wanted to say that I read your post and am not ignoring you
no problem. I don't think I'll ever use those, anyway, but it's an interesting problem (methinks).

@Clou: How do S3Fox-created "European" buckets look like when listing them with s3cmd listbuckets?
Logged
Clou
Newbie
*
Posts: 10


View Profile
« Reply #6 on: November 07, 2007, 11:48:14 AM »

I don't have one of those buckets but I'm guessing that the server name might be different?  You can set that via the env variable AWS_S3_HOST (defaults to s3.amazonaws.com)

I can confirm that. Debug output shows that a request is expected to <bucketname>.s3.amazonaws.com. Setting the env-variable to that hostname leads to another error:

<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>...

So there might be something wrong with request signing.

@maelcum: The buckets don't look different when viewed via listbuckets...
Logged
Clou
Newbie
*
Posts: 10


View Profile
« Reply #7 on: November 08, 2007, 08:50:08 AM »

I think I got behind what might be the problem: The permanent redirect is to <bucketname>.s3.amazonaws.com, and so the bucketname is encoded into the hostname.
So the request is made to
<bucketname>.s3.amazonaws.com/<bucketname>
which is wrong and should be
<bucketname>.s3.amazonaws.com/

(just a wild guess...)
Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #8 on: November 08, 2007, 04:58:53 PM »

Last I looked their API docs haven't been updated.  Anyone know if there's some official way to deal with this?  I'd rather not code based on empirical results.
Logged
Clou
Newbie
*
Posts: 10


View Profile
« Reply #9 on: November 08, 2007, 05:18:08 PM »

The API Docs were last updated Nov-5. The relevant section seems to be "Location Selection" where it is mentioned:

====
To access Amazon S3 buckets and objects that were created using CreateBucketConfiguration, you must use the virtual hosted-style request. For example:
http://yourbucket.s3.amazonaws.com/yourobject
You cannot use the path-style request:
http://s3.amazonaws.com/yourbucket/yourobject
If you use the path-style request, you receive a permanent redirect.
====

Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #10 on: November 09, 2007, 03:56:22 PM »

Thanks for the info.. They updated it like THE VERY DAY after I checked.
Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #11 on: November 09, 2007, 04:49:50 PM »

Feel like an idiot but I still see the 2006 version:
http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=48

Am I missing something?  Where is the updated one?
Logged
Clou
Newbie
*
Posts: 10


View Profile
« Reply #12 on: November 09, 2007, 05:37:26 PM »

The API Version is still from 2006, but the developer guide was last updated Nov-5. There you can find information on the location constraints.

HTH
Clou
Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #13 on: November 10, 2007, 09:23:03 PM »

Link I find to the dev guide is: http://developer.amazonwebservices.com/connect/entry.jspa?entryID=123
which still says 2006.  Halp?  Can you post the link?
Logged
Clou
Newbie
*
Posts: 10


View Profile
« Reply #14 on: November 12, 2007, 03:16:45 AM »

The Link is
http://docs.amazonwebservices.com/AmazonS3/2006-03-01/

Don't mind the date in the URL, which means that the API version has not changed. Obviously when the new functionality was implemented it was not neccessary to change the API itself...
Have a look at the "What's new" section...
« Last Edit: November 12, 2007, 08:02:36 AM by Clou » Logged
Pages: [1] 2 3 4
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2013, Simple Machines Valid XHTML 1.0! Valid CSS!