S3Sync.net

General Category => Questions => Topic started by: maelcum on November 06, 2007, 09:18:18 AM



Title: How to choose storage location in Amazon S3
Post by: maelcum on November 06, 2007, 09:18:18 AM
Hi,

not a s3sync/s3cmd-question, but concerning data storage in Amazon S3...

Since there is a choice to store data in the US or Europe, I'm trying to find out how this could be done.
So far I came across a FAQ which says that this decision is made when creating a bucket. I haven't found out how I have to name my bucket, though, to make sure its in the US (or Europe - where it is more expensive btw).

Could somebody point me into the right direction?

Thanks very much.

Cheers

maelcum


Title: Re: How to choose storage location in Amazon S3
Post by: ferrix on November 06, 2007, 02:32:56 PM
I don't know the answer, but I just wanted to say that I read your post and am not ignoring you :D :D


Title: Re: How to choose storage location in Amazon S3
Post by: PolarLava on November 06, 2007, 03:22:54 PM
"To support Europe we enhanced Amazon S3’s API to include a new parameter for locality. The name of the parameter is LocationConstraint, and if you use it then buckets are created in the location you specify. Existing applications will not be affected, and will create buckets in the United States, because that’s the “default” behavior. Pricing is slightly different in Europe; although not but much. Use the AWS Price calculator if you want to estimate what your costs will be."

This comes from the AWS Blog http://aws.typepad.com/aws/2007/11/notes-from-berl.html (http://aws.typepad.com/aws/2007/11/notes-from-berl.html)

I would assume this will require some change to S3Sync to support selecting whether the bucket is created in the US or Europe.

-klp


Title: Re: How to choose storage location in Amazon S3
Post by: Clou on November 07, 2007, 05:16:40 AM
I created a "european" bucket with S3Fox, then tried to fill it with s3sync.rb.
Filling failed with:



./s3sync.rb --verbose /var/archives/ testeu:test
S3 command failed:
list_bucket testeu max-keys 200 prefix test delimiter /
With result 301 Moved Permanently
S3 ERROR: #<Net::HTTPMovedPermanently:0xb7ad27a0>
./s3sync.rb:292:in `+': can't convert nil into Array (TypeError)
        from ./s3sync.rb:292:in `s3TreeRecurse'
        from ./s3sync.rb:348:in `main'
        from ./thread_generator.rb:79:in `call'
        from ./thread_generator.rb:79:in `initialize'
        from ./thread_generator.rb:76:in `new'
        from ./thread_generator.rb:76:in `initialize'
        from ./s3sync.rb:269:in `new'
        from ./s3sync.rb:269:in `main'
        from ./s3sync.rb:708


Title: Re: How to choose storage location in Amazon S3
Post by: ferrix on November 07, 2007, 10:05:39 AM
I don't have one of those buckets but I'm guessing that the server name might be different?  You can set that via the env variable AWS_S3_HOST (defaults to s3.amazonaws.com)


Title: Re: How to choose storage location in Amazon S3
Post by: maelcum on November 07, 2007, 10:15:18 AM

@Ferrix:
Quote
I don't know the answer, but I just wanted to say that I read your post and am not ignoring you
no problem. I don't think I'll ever use those, anyway, but it's an interesting problem (methinks).

@Clou: How do S3Fox-created "European" buckets look like when listing them with s3cmd listbuckets?


Title: Re: How to choose storage location in Amazon S3
Post by: Clou on November 07, 2007, 11:48:14 AM
I don't have one of those buckets but I'm guessing that the server name might be different?  You can set that via the env variable AWS_S3_HOST (defaults to s3.amazonaws.com)

I can confirm that. Debug output shows that a request is expected to <bucketname>.s3.amazonaws.com. Setting the env-variable to that hostname leads to another error:

<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>...

So there might be something wrong with request signing.

@maelcum: The buckets don't look different when viewed via listbuckets...


Title: Re: How to choose storage location in Amazon S3
Post by: Clou on November 08, 2007, 08:50:08 AM
I think I got behind what might be the problem: The permanent redirect is to <bucketname>.s3.amazonaws.com, and so the bucketname is encoded into the hostname.
So the request is made to
<bucketname>.s3.amazonaws.com/<bucketname>
which is wrong and should be
<bucketname>.s3.amazonaws.com/

(just a wild guess...)


Title: Re: How to choose storage location in Amazon S3
Post by: ferrix on November 08, 2007, 04:58:53 PM
Last I looked their API docs haven't been updated.  Anyone know if there's some official way to deal with this?  I'd rather not code based on empirical results.


Title: Re: How to choose storage location in Amazon S3
Post by: Clou on November 08, 2007, 05:18:08 PM
The API Docs were last updated Nov-5. The relevant section seems to be "Location Selection" where it is mentioned:

====
To access Amazon S3 buckets and objects that were created using CreateBucketConfiguration, you must use the virtual hosted-style request. For example:
http://yourbucket.s3.amazonaws.com/yourobject
You cannot use the path-style request:
http://s3.amazonaws.com/yourbucket/yourobject
If you use the path-style request, you receive a permanent redirect.
====



Title: Re: How to choose storage location in Amazon S3
Post by: ferrix on November 09, 2007, 03:56:22 PM
Thanks for the info.. They updated it like THE VERY DAY after I checked.


Title: Re: How to choose storage location in Amazon S3
Post by: ferrix on November 09, 2007, 04:49:50 PM
Feel like an idiot but I still see the 2006 version:
http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=48

Am I missing something?  Where is the updated one?


Title: Re: How to choose storage location in Amazon S3
Post by: Clou on November 09, 2007, 05:37:26 PM
The API Version is still from 2006, but the developer guide was last updated Nov-5. There you can find information on the location constraints.

HTH
Clou


Title: Re: How to choose storage location in Amazon S3
Post by: ferrix on November 10, 2007, 09:23:03 PM
Link I find to the dev guide is: http://developer.amazonwebservices.com/connect/entry.jspa?entryID=123
which still says 2006.  Halp?  Can you post the link?


Title: Re: How to choose storage location in Amazon S3
Post by: Clou on November 12, 2007, 03:16:45 AM
The Link is
http://docs.amazonwebservices.com/AmazonS3/2006-03-01/

Don't mind the date in the URL, which means that the API version has not changed. Obviously when the new functionality was implemented it was not neccessary to change the API itself...
Have a look at the "What's new" section...


Title: Re: How to choose storage location in Amazon S3
Post by: ferrix on November 12, 2007, 01:19:00 PM
Wow holding that date constant just totally messed with me.  Thanks for the help!


Title: Re: How to choose storage location in Amazon S3
Post by: ferrix on November 12, 2007, 02:18:30 PM
OK, this is going to actually be a big deal.  The underlying aws ruby code changed a bunch.  working on it...


Title: Re: How to choose storage location in Amazon S3
Post by: jckdnk111 on November 13, 2007, 11:48:04 AM
I'm glad you are on this so quickly ... I've been fighting with it for 2 days and still no luck.
I'm eagerly awaiting the fix.
Thanks for continuing to support your excellent application!


Title: Re: How to choose storage location in Amazon S3
Post by: maxi on November 13, 2007, 12:06:48 PM
Hi there!

exactly the same problem here. It seemed that EC2 instances that are already running for quite a time still work. But newly created instanced first get a 301 Moved Permanently and then (after finding out that aws_s3_host: <bucketname>.s3.amazonaws.com should be used) deny any access :'-(

Looking forward to a fix :-) Regards & thanks for the great work!!

Maxi


Title: Re: How to choose storage location in Amazon S3
Post by: mrvanes on November 15, 2007, 08:31:49 AM
Just 5 minutes into trying s3sync I stumbled at exactly this problem and as far as I understand the fix is quite simple.
The only thing you need to do is change your URL scheme

from http://s3.amazonaws.com/<bucketname>
to http://<bucketname>.s3.amazonaws.com/

and DNS will do the rest... it's transparent, it's easy. As far as I understand.

I would change it myself if I knew ruby and where you construct your URL in the code.


Title: Re: How to choose storage location in Amazon S3
Post by: mrvanes on November 15, 2007, 08:42:40 AM
Ok, so a quick glance learned you construct the complete path (bucket+key) outside generate_url. Too bad, this requires a lot more work now, since you need the bucket and the path separately when generating the URL for the above trick to work... :(


Title: Re: How to choose storage location in Amazon S3
Post by: maxi on November 15, 2007, 08:58:18 AM
Yeah - actually I thought that, too but since ferrix wrote "OK, this is going to actually be a big deal." I wasn't too optimistic though :-}. Unfortunately I'm a ruby n00b as well and therefor I'd highly appreciate any workarounds for this problem. Anyone has an Idea yet?

Regards - Maxi


Title: Re: How to choose storage location in Amazon S3
Post by: ferrix on November 15, 2007, 04:04:59 PM
I'm going to try to get to this this weekend.


Title: Re: How to choose storage location in Amazon S3
Post by: maxi on November 16, 2007, 05:09:21 AM
Great! I'm looking forward to it :-)


Title: Re: How to choose storage location in Amazon S3
Post by: Clou on November 16, 2007, 06:19:14 AM
Just 5 minutes into trying s3sync I stumbled at exactly this problem and as far as I understand the fix is quite simple.
The only thing you need to do is change your URL scheme

from http://s3.amazonaws.com/<bucketname>
to http://<bucketname>.s3.amazonaws.com/

and DNS will do the rest... it's transparent, it's easy. As far as I understand.

I would change it myself if I knew ruby and where you construct your URL in the code.

Unfortunately it's not that simple: You could use the changed URL scheme, and it would work for all EU buckets, and for most US buckets. But in US buckets uppercase letters are allowed, and those buckets can not be addressed as virtual hosts... I dont' know if there are other characters that can be used in US buckets which cannot be used in EU buckets. But that will have to be considered.


I'm also not very firm in ruby and don't know how to code it, so i'm looking forward to see when ferrix has had time to look into it.


Title: Re: How to choose storage location in Amazon S3
Post by: maxi on November 16, 2007, 06:24:50 AM
@Clou: You're mentioning an interesting point: There are US and EU Buckets. But for now (as far as I could see) there are only EC2-Instances for US I guess. So when combining EC2/S3 for now it would only make sense to have US-Only EC2/S3-Combinations unless one want to give EU-Customers direct access to S3-Buckets via Link, right?

Regards - Maxi


Title: Re: How to choose storage location in Amazon S3
Post by: Clou on November 16, 2007, 06:53:08 AM
@Clou: You're mentioning an interesting point: There are US and EU Buckets. But for now (as far as I could see) there are only EC2-Instances for US I guess. So when combining EC2/S3 for now it would only make sense to have US-Only EC2/S3-Combinations unless one want to give EU-Customers direct access to S3-Buckets via Link, right?

As I understand it that's correct. It also has to be considered that traffic is charged between EC2/S3-EU but not between EC2/S3-US, but we're getting off-topic ;-)


Title: Re: How to choose storage location in Amazon S3
Post by: ferrix on November 17, 2007, 08:25:11 PM
New release on its way; check announce board.


Title: Re: How to choose storage location in Amazon S3
Post by: maxi on November 18, 2007, 08:57:04 AM
I just downloaded/installed/tried it. It worked immediately ;-) Great! Thanks a lot!!

Regards - Maxi


Title: Re: How to choose storage location in Amazon S3
Post by: maxi on November 18, 2007, 09:15:56 AM
hmm - now that I tried another bucket there's still the old problem again :-( Any Idea?

Response code: 403
S3 command failed:
put test #<S3::S3Object:0xb7cebbe4> Content-Length 13240
With result 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><RequestId>0B42B62520D73B5B</RequestId><SignatureProvided>6WBByPaheeQLJx3nCrqkPAbkWCs=</SignatureProvided><StringToSignBytes>50 55 54 0a 0a 0a 53 75 6e 2c 20 31 38 20 4e 6f 76 20


Title: Re: How to choose storage location in Amazon S3
Post by: maxi on November 18, 2007, 09:24:36 AM
yet me again - sorry!

I guess I found the problem (although I don't understand it) - creating the bucket using S3Sync somehow helps. After that it works.. Obiously the error only comes up when creating the bucket with e.g. the s3fox firefox plugin. Then it gets listed when doing a listbucket but produces the error mentioned above when trying to put files on.

Regards - maxi


Title: Re: How to choose storage location in Amazon S3
Post by: Clou on November 18, 2007, 03:29:40 PM
Excellent work, it works for me.
But there seems to be a small typo in README.txt: The environment variable to choose the calling format -needed for european buckets- seems to be AWS_CALLING_FORMAT instead of S3SYNC_CALLING_FORMAT.

Greets,
Clou


Title: Re: How to choose storage location in Amazon S3
Post by: ferrix on November 18, 2007, 04:43:56 PM
yet me again - sorry!

I guess I found the problem (although I don't understand it) - creating the bucket using S3Sync somehow helps. After that it works.. Obiously the error only comes up when creating the bucket with e.g. the s3fox firefox plugin. Then it gets listed when doing a listbucket but produces the error mentioned above when trying to put files on.

Regards - maxi

Tell me the command line you're using and the rest of the error message and I might be able to provide insight.


Title: Re: How to choose storage location in Amazon S3
Post by: ferrix on November 18, 2007, 04:47:22 PM
Excellent work, it works for me.
But there seems to be a small typo in README.txt: The environment variable to choose the calling format -needed for european buckets- seems to be AWS_CALLING_FORMAT instead of S3SYNC_CALLING_FORMAT.

Greets,
Clou

Fixed thanks!


Title: Re: How to choose storage location in Amazon S3
Post by: mrvanes on November 19, 2007, 10:59:00 AM
Thx for the extremely quick fix!
I'll try it tonight but do I understand correctly this "feature" won't be transparent? It seems s3fox is able to handle it transparent, but it might of course keep track of the location internally?


Title: Re: How to choose storage location in Amazon S3
Post by: maelcum on November 19, 2007, 01:16:18 PM
Wohooo! Works for me too!!

Haven't stress-tested it yet (afraid of loosing something) but synced some thousand files and had no problem.

Thanks ferrix. You're great.


Title: Re: How to choose storage location in Amazon S3
Post by: maelcum on November 19, 2007, 02:01:57 PM
While you are at fixing the Readme:
Quote
s3sync runs happily on linux, probably other *ix, and also Windows (except that symlinks and permissions management features don't do anything on Windows). If you get it running somewhere interesting let me know (see below)

It runs on Macs quite nicely. Although you'd have to update the ruby library. But then - you'd use the same thing as on Windows: Ruby One-Click Installer: http://rubyosx.rubyforge.org/ (http://rubyosx.rubyforge.org/)

 ;D


Title: Re: How to choose storage location in Amazon S3
Post by: mrvanes on November 19, 2007, 03:37:00 PM
Sorry, but I have to fail report on ALL attempts.
I too, use s3fox to administrate my buckets, but a EU bucket created in s3fox is not accessible with s3sync:

Code:
$ ./s3sync.rb -vrnd /home/martin/test mrvanes-s3fox-eu:pre
s3Prefix pre/test
localPrefix /home/martin/test
local node object init. Name: Path:/home/martin/test/ Size:38 Tag:d66759af42f282e1ba19144df2d405d0
s3TreeRecurse mrvanes-s3fox-eu pre/test
Creating new connection
Trying command list_bucket mrvanes-s3fox-eu max-keys 200 prefix pre/test delimiter / with 100 retries left
Response code: 403
S3 command failed:
list_bucket max-keys 200 prefix pre/test delimiter /
With result 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><RequestId>xxx</RequestId><SignatureProvided>xxx=</SignatureProvided><StringToSignBytes>47 45 54 0a 0a 0a 4d 6f 6e 2c 20 31 39 20 4e 6f 76 20 32 30 30 37 20 32 30 3a 33 35 3a 32 33 20 47 4d 54 0a 2f</StringToSignBytes><AWSAccessKeyId>xxx</AWSAccessKeyId><HostId>xxx</HostId><StringToSign>GET


Mon, 19 Nov 2007 20:35:23 GMT
/</StringToSign></Error>
S3 ERROR: #<Net::HTTPForbidden:0xb7c2d848>
./s3sync.rb:292:in `+': can't convert nil into Array (TypeError)
        from ./s3sync.rb:292:in `s3TreeRecurse'
        from ./s3sync.rb:348:in `main'
        from ./thread_generator.rb:79:in `call'
        from ./thread_generator.rb:79:in `initialize'
        from ./thread_generator.rb:76:in `new'
        from ./thread_generator.rb:76:in `initialize'
        from ./s3sync.rb:269:in `new'
        from ./s3sync.rb:269:in `main'
        from ./s3sync.rb:711

I get exactly the same errors on buckets created wich s3cmd btw, so that doesn't seem to help me.

On a sidenote, s3cmd doesn't seem to play nice with aws_calling_format, setting it to subdomain renders s3cmd useless, so I need to comment it whenever I use s3cmd on european buckets (create/list/delete).

So, so far, I haven't been able to successfully store anything on European s3.

I cleared my output from any secret looking info, but if it's helpfull to print everything and someone can convince me it's safe I'm more than willing to cooperate ;)


Title: Re: How to choose storage location in Amazon S3
Post by: mrvanes on November 20, 2007, 05:09:20 AM
Found the error: I stubbornly set the AWS_CALLING_FORMAT variable in s3config.yml just like the others like this:
aws_calling_format: subdomain

but this doesn't seem to work. Now I explicitly exported AWS_CALLING_FORMAT=SUBDOMAIN and my european bucket gets filled!

Strange thing is dat s3cmd seems to react to the yml setting because it matters if I use the setting or not!


Title: Re: How to choose storage location in Amazon S3
Post by: mrvanes on November 20, 2007, 05:31:19 AM
Just to be as complete as possible, a european bucket created with s3fox returns this error:

Code:
S3 command failed:
list_bucket max-keys 200 prefix pre/test delimiter /
With result 307 Temporary Redirect
S3 ERROR: #<Net::HTTPTemporaryRedirect:0xb7b7f52c>
./s3sync.rb:292:in `+': can't convert nil into Array (TypeError)
        from ./s3sync.rb:292:in `s3TreeRecurse'
        from ./s3sync.rb:348:in `main'
        from ./thread_generator.rb:79:in `call'
        from ./thread_generator.rb:79:in `initialize'
        from ./thread_generator.rb:76:in `new'
        from ./thread_generator.rb:76:in `initialize'
        from ./s3sync.rb:269:in `new'
        from ./s3sync.rb:269:in `main'
        from ./s3sync.rb:711

s3cmd location <bucket> returns EU.

Edit: This is why: http://developer.amazonwebservices.com/connect/message.jspa?messageID=71360#71360

It's all in DNS... :-/


Title: Re: How to choose storage location in Amazon S3
Post by: ferrix on November 20, 2007, 07:32:16 PM
aws_calling_format: subdomain

Case sensitive value, SUBDOMAIN!  I don't use or test the yaml.  If there's a bug someone can point at I'll have a go at fixing it.


Title: Re: How to choose storage location in Amazon S3
Post by: ferrix on November 20, 2007, 07:33:35 PM
Just to be as complete as possible, a european bucket created with s3fox returns this error:

Code:
S3 command failed:
list_bucket max-keys 200 prefix pre/test delimiter /
With result 307 Temporary Redirect
...

It's all in DNS... :-/

I guess this 307 case needs to be handled; I'll try to figure it out.


Title: Re: How to choose storage location in Amazon S3
Post by: ferrix on November 20, 2007, 09:39:54 PM
Augh! The ruby S3 lib totally doesn't do the stuff I need to make this work =/

It's not just the 307's, I have to figure out how to make the PUTs do 100-continue responses.


Title: Re: How to choose storage location in Amazon S3
Post by: ferrix on November 21, 2007, 12:20:37 AM
Done in 1.2.2

I still can't do 100-continue (I'd have to ditch the ruby http lib) but it shouldn't matter in practice.  For s3sync the first operation is always a list, so any redirect will be found out there.

For s3cmd the lack of 100-continue support means there is a time window on brand new EU buckets (several minutes after creation in my tests) where a PUT will have to transfer the data twice.  Once to the default host, then again after the 307 redirect.

Phew!  I'm done for a while.


Title: Re: How to choose storage location in Amazon S3
Post by: mrvanes on November 21, 2007, 03:31:57 AM
aws_calling_format: subdomain

Case sensitive value, SUBDOMAIN!  I don't use or test the yaml.  If there's a bug someone can point at I'll have a go at fixing it.

I tried capitals but that didn't work. Will double check later.

update: checked and confirmed: aws_calling_format: SUBDOMAIN in s3config.yml yields a 403 (forbidden, Error: SignatureDoesNotMatch)


Title: Re: How to choose storage location in Amazon S3
Post by: denix on December 11, 2007, 05:54:35 AM
update: checked and confirmed: aws_calling_format: SUBDOMAIN in s3config.yml yields a 403 (forbidden, Error: SignatureDoesNotMatch)

I have this problem also.

This works:
AWS_CALLING_FORMAT=SUBDOMAIN ./s3cmd.rb createbucket mybucket EU
but only if I remove
AWS_CALLING_FORMAT: SUBDOMAIN
from s3config.yml. Otherwise it fails with 403 Forbidden (SignatureDoesNotMatch)


Title: Re: How to choose storage location in Amazon S3
Post by: ferrix on January 06, 2008, 09:52:58 AM

I have this problem also.

This works:
AWS_CALLING_FORMAT=SUBDOMAIN ./s3cmd.rb createbucket mybucket EU
but only if I remove
AWS_CALLING_FORMAT: SUBDOMAIN
from s3config.yml. Otherwise it fails with 403 Forbidden (SignatureDoesNotMatch)

This will be fixed today; watch announce.