How to choose storage location in Amazon S3

[maxi]

yet me again - sorry!

I guess I found the problem (although I don't understand it) - creating the bucket using S3Sync somehow helps. After that it works.. Obiously the error only comes up when creating the bucket with e.g. the s3fox firefox plugin. Then it gets listed when doing a listbucket but produces the error mentioned above when trying to put files on.

Regards - maxi

[Clou]

Excellent work, it works for me.
But there seems to be a small typo in README.txt: The environment variable to choose the calling format -needed for european buckets- seems to be AWS_CALLING_FORMAT instead of S3SYNC_CALLING_FORMAT.

Greets,
Clou

[ferrix]

yet me again - sorry!

I guess I found the problem (although I don't understand it) - creating the bucket using S3Sync somehow helps. After that it works.. Obiously the error only comes up when creating the bucket with e.g. the s3fox firefox plugin. Then it gets listed when doing a listbucket but produces the error mentioned above when trying to put files on.

Regards - maxi

Tell me the command line you're using and the rest of the error message and I might be able to provide insight.

[ferrix]

Excellent work, it works for me.
But there seems to be a small typo in README.txt: The environment variable to choose the calling format -needed for european buckets- seems to be AWS_CALLING_FORMAT instead of S3SYNC_CALLING_FORMAT.

Greets,
Clou

Fixed thanks!

[mrvanes]

Thx for the extremely quick fix!
I'll try it tonight but do I understand correctly this "feature" won't be transparent? It seems s3fox is able to handle it transparent, but it might of course keep track of the location internally?

[maelcum]

Wohooo! Works for me too!!

Haven't stress-tested it yet (afraid of loosing something) but synced some thousand files and had no problem.

Thanks ferrix. You're great.

[maelcum]

While you are at fixing the Readme:

s3sync runs happily on linux, probably other *ix, and also Windows (except that symlinks and permissions management features don't do anything on Windows). If you get it running somewhere interesting let me know (see below)

It runs on Macs quite nicely. Although you'd have to update the ruby library. But then - you'd use the same thing as on Windows: Ruby One-Click Installer: http://rubyosx.rubyforge.org/

 

[mrvanes]

Sorry, but I have to fail report on ALL attempts.
I too, use s3fox to administrate my buckets, but a EU bucket created in s3fox is not accessible with s3sync:

Code:

$ ./s3sync.rb -vrnd /home/martin/test mrvanes-s3fox-eu:pre
s3Prefix pre/test
localPrefix /home/martin/test
local node object init. Name: Path:/home/martin/test/ Size:38 Tag:d66759af42f282e1ba19144df2d405d0
s3TreeRecurse mrvanes-s3fox-eu pre/test
Creating new connection
Trying command list_bucket mrvanes-s3fox-eu max-keys 200 prefix pre/test delimiter / with 100 retries left
Response code: 403
S3 command failed:
list_bucket max-keys 200 prefix pre/test delimiter /
With result 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><RequestId>xxx</RequestId><SignatureProvided>xxx=</SignatureProvided><StringToSignBytes>47 45 54 0a 0a 0a 4d 6f 6e 2c 20 31 39 20 4e 6f 76 20 32 30 30 37 20 32 30 3a 33 35 3a 32 33 20 47 4d 54 0a 2f</StringToSignBytes><AWSAccessKeyId>xxx</AWSAccessKeyId><HostId>xxx</HostId><StringToSign>GET


Mon, 19 Nov 2007 20:35:23 GMT
/</StringToSign></Error>
S3 ERROR: #<Net::HTTPForbidden:0xb7c2d848>
./s3sync.rb:292:in `+': can't convert nil into Array (TypeError)
        from ./s3sync.rb:292:in `s3TreeRecurse'
        from ./s3sync.rb:348:in `main'
        from ./thread_generator.rb:79:in `call'
        from ./thread_generator.rb:79:in `initialize'
        from ./thread_generator.rb:76:in `new'
        from ./thread_generator.rb:76:in `initialize'
        from ./s3sync.rb:269:in `new'
        from ./s3sync.rb:269:in `main'
        from ./s3sync.rb:711

I get exactly the same errors on buckets created wich s3cmd btw, so that doesn't seem to help me.

On a sidenote, s3cmd doesn't seem to play nice with aws_calling_format, setting it to subdomain renders s3cmd useless, so I need to comment it whenever I use s3cmd on european buckets (create/list/delete).

So, so far, I haven't been able to successfully store anything on European s3.

I cleared my output from any secret looking info, but if it's helpfull to print everything and someone can convince me it's safe I'm more than willing to cooperate

[mrvanes]

Found the error: I stubbornly set the AWS_CALLING_FORMAT variable in s3config.yml just like the others like this:
aws_calling_format: subdomain

but this doesn't seem to work. Now I explicitly exported AWS_CALLING_FORMAT=SUBDOMAIN and my european bucket gets filled!

Strange thing is dat s3cmd seems to react to the yml setting because it matters if I use the setting or not!

[mrvanes]

Just to be as complete as possible, a european bucket created with s3fox returns this error:

Code:

S3 command failed:
list_bucket max-keys 200 prefix pre/test delimiter /
With result 307 Temporary Redirect
S3 ERROR: #<Net::HTTPTemporaryRedirect:0xb7b7f52c>
./s3sync.rb:292:in `+': can't convert nil into Array (TypeError)
        from ./s3sync.rb:292:in `s3TreeRecurse'
        from ./s3sync.rb:348:in `main'
        from ./thread_generator.rb:79:in `call'
        from ./thread_generator.rb:79:in `initialize'
        from ./thread_generator.rb:76:in `new'
        from ./thread_generator.rb:76:in `initialize'
        from ./s3sync.rb:269:in `new'
        from ./s3sync.rb:269:in `main'
        from ./s3sync.rb:711

s3cmd location <bucket> returns EU.

Edit: This is why: http://developer.amazonwebservices.com/connect/message.jspa?messageID=71360#71360

It's all in DNS... :-/

[ferrix]

aws_calling_format: subdomain

Case sensitive value, SUBDOMAIN!  I don't use or test the yaml.  If there's a bug someone can point at I'll have a go at fixing it.

[ferrix]

Just to be as complete as possible, a european bucket created with s3fox returns this error:

Code:

S3 command failed:
list_bucket max-keys 200 prefix pre/test delimiter /
With result 307 Temporary Redirect
...

It's all in DNS... :-/

I guess this 307 case needs to be handled; I'll try to figure it out.

[ferrix]

Augh! The ruby S3 lib totally doesn't do the stuff I need to make this work =/

It's not just the 307's, I have to figure out how to make the PUTs do 100-continue responses.

[ferrix]

Done in 1.2.2

I still can't do 100-continue (I'd have to ditch the ruby http lib) but it shouldn't matter in practice.  For s3sync the first operation is always a list, so any redirect will be found out there.

For s3cmd the lack of 100-continue support means there is a time window on brand new EU buckets (several minutes after creation in my tests) where a PUT will have to transfer the data twice.  Once to the default host, then again after the 307 redirect.

Phew!  I'm done for a while.

[mrvanes]

aws_calling_format: subdomain

Case sensitive value, SUBDOMAIN!  I don't use or test the yaml.  If there's a bug someone can point at I'll have a go at fixing it.

I tried capitals but that didn't work. Will double check later.

update: checked and confirmed: aws_calling_format: SUBDOMAIN in s3config.yml yields a 403 (forbidden, Error: SignatureDoesNotMatch)