How to choose storage location in Amazon S3

[maelcum]

Hi,

not a s3sync/s3cmd-question, but concerning data storage in Amazon S3...

Since there is a choice to store data in the US or Europe, I'm trying to find out how this could be done.
So far I came across a FAQ which says that this decision is made when creating a bucket. I haven't found out how I have to name my bucket, though, to make sure its in the US (or Europe - where it is more expensive btw).

Could somebody point me into the right direction?

Thanks very much.

Cheers

maelcum

[ferrix]

I don't know the answer, but I just wanted to say that I read your post and am not ignoring you

[PolarLava]

"To support Europe we enhanced Amazon S3’s API to include a new parameter for locality. The name of the parameter is LocationConstraint, and if you use it then buckets are created in the location you specify. Existing applications will not be affected, and will create buckets in the United States, because that’s the “default” behavior. Pricing is slightly different in Europe; although not but much. Use the AWS Price calculator if you want to estimate what your costs will be."

This comes from the AWS Blog http://aws.typepad.com/aws/2007/11/notes-from-berl.html

I would assume this will require some change to S3Sync to support selecting whether the bucket is created in the US or Europe.

-klp

[Clou]

I created a "european" bucket with S3Fox, then tried to fill it with s3sync.rb.
Filling failed with:



./s3sync.rb --verbose /var/archives/ testeu:test
S3 command failed:
list_bucket testeu max-keys 200 prefix test delimiter /
With result 301 Moved Permanently
S3 ERROR: #<Net::HTTPMovedPermanently:0xb7ad27a0>
./s3sync.rb:292:in `+': can't convert nil into Array (TypeError)
        from ./s3sync.rb:292:in `s3TreeRecurse'
        from ./s3sync.rb:348:in `main'
        from ./thread_generator.rb:79:in `call'
        from ./thread_generator.rb:79:in `initialize'
        from ./thread_generator.rb:76:in `new'
        from ./thread_generator.rb:76:in `initialize'
        from ./s3sync.rb:269:in `new'
        from ./s3sync.rb:269:in `main'
        from ./s3sync.rb:708

[ferrix]

I don't have one of those buckets but I'm guessing that the server name might be different?  You can set that via the env variable AWS_S3_HOST (defaults to s3.amazonaws.com)

[maelcum]

@Ferrix:

I don't know the answer, but I just wanted to say that I read your post and am not ignoring you

no problem. I don't think I'll ever use those, anyway, but it's an interesting problem (methinks).

@Clou: How do S3Fox-created "European" buckets look like when listing them with s3cmd listbuckets?

[Clou]

I don't have one of those buckets but I'm guessing that the server name might be different?  You can set that via the env variable AWS_S3_HOST (defaults to s3.amazonaws.com)

I can confirm that. Debug output shows that a request is expected to <bucketname>.s3.amazonaws.com. Setting the env-variable to that hostname leads to another error:

<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>...

So there might be something wrong with request signing.

@maelcum: The buckets don't look different when viewed via listbuckets...

[Clou]

I think I got behind what might be the problem: The permanent redirect is to <bucketname>.s3.amazonaws.com, and so the bucketname is encoded into the hostname.
So the request is made to
<bucketname>.s3.amazonaws.com/<bucketname>
which is wrong and should be
<bucketname>.s3.amazonaws.com/

(just a wild guess...)

[ferrix]

Last I looked their API docs haven't been updated.  Anyone know if there's some official way to deal with this?  I'd rather not code based on empirical results.

[Clou]

The API Docs were last updated Nov-5. The relevant section seems to be "Location Selection" where it is mentioned:

====
To access Amazon S3 buckets and objects that were created using CreateBucketConfiguration, you must use the virtual hosted-style request. For example:
http://yourbucket.s3.amazonaws.com/yourobject
You cannot use the path-style request:
http://s3.amazonaws.com/yourbucket/yourobject
If you use the path-style request, you receive a permanent redirect.
====

[ferrix]

Thanks for the info.. They updated it like THE VERY DAY after I checked.

[ferrix]

Feel like an idiot but I still see the 2006 version:
http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=48

Am I missing something?  Where is the updated one?

[Clou]

The API Version is still from 2006, but the developer guide was last updated Nov-5. There you can find information on the location constraints.

HTH
Clou

[ferrix]

Link I find to the dev guide is: http://developer.amazonwebservices.com/connect/entry.jspa?entryID=123
which still says 2006.  Halp?  Can you post the link?

[Clou]

The Link is
http://docs.amazonwebservices.com/AmazonS3/2006-03-01/

Don't mind the date in the URL, which means that the API version has not changed. Obviously when the new functionality was implemented it was not neccessary to change the API itself...
Have a look at the "What's new" section...