I have been thinking about this and thought this might have to do with buffering. This could be the responsibility/flaw of the shell (bash), but also Ruby. Is there a way to flush buffers in Ruby, like there is in PHP?

This is funny.
Since the backup takes a lot of time, but I want to monitor the backup process at any given time I redirect stdout and stderr to 2 files like this in the backup script.

./s3sync.rb -rv /media/fotos [somebucket]: >backup.log 2>backup.err

I noticed that the process seemed to stop for no apparent reason (backup.log didn't grow) so I restarted it a couple of times and I was thinking about a bugreport or not untill I extended the stderr output with -d!

./s3sync.rb -rvd /media/fotos [somebucket]: >backup.log 2>backup.err

Now this is interresting, because I can monitor progress of s3sync in backup.err (and I see it does!) but backup.log suddenly stops! That's when I saw the filesize of both redirect files:

4364 -rw-r--r--  1 root root  4452913 Nov 22 15:55 backup.err
  16 -rw-r--r--  1 root root    16384 Nov 22 15:41 backup.log

16384 is exactly 16K

I would think this had something to do with my shell or ruby, but stderr continues way beyond 16K so I suspect a hard limit somewhere in s3sync?
Anyway, I thought it would be worth mentioning it.

update: It doesn't max out on 16K, it just stalls a _very_ long time on 16K...

Martin

aws_calling_format: subdomain

Case sensitive value, SUBDOMAIN!  I don't use or test the yaml.  If there's a bug someone can point at I'll have a go at fixing it.

I tried capitals but that didn't work. Will double check later.

update: checked and confirmed: aws_calling_format: SUBDOMAIN in s3config.yml yields a 403 (forbidden, Error: SignatureDoesNotMatch)

Just to be as complete as possible, a european bucket created with s3fox returns this error:

Code:

S3 command failed:
list_bucket max-keys 200 prefix pre/test delimiter /
With result 307 Temporary Redirect
S3 ERROR: #<Net::HTTPTemporaryRedirect:0xb7b7f52c>
./s3sync.rb:292:in `+': can't convert nil into Array (TypeError)
        from ./s3sync.rb:292:in `s3TreeRecurse'
        from ./s3sync.rb:348:in `main'
        from ./thread_generator.rb:79:in `call'
        from ./thread_generator.rb:79:in `initialize'
        from ./thread_generator.rb:76:in `new'
        from ./thread_generator.rb:76:in `initialize'
        from ./s3sync.rb:269:in `new'
        from ./s3sync.rb:269:in `main'
        from ./s3sync.rb:711

s3cmd location <bucket> returns EU.

Edit: This is why: http://developer.amazonwebservices.com/connect/message.jspa?messageID=71360#71360

It's all in DNS... :-/

Found the error: I stubbornly set the AWS_CALLING_FORMAT variable in s3config.yml just like the others like this:
aws_calling_format: subdomain

but this doesn't seem to work. Now I explicitly exported AWS_CALLING_FORMAT=SUBDOMAIN and my european bucket gets filled!

Strange thing is dat s3cmd seems to react to the yml setting because it matters if I use the setting or not!

I'm really just scratching the surface of s3sync and hit a stupid failure when I tried to upload a test directory into the prefix that exactly resembled the source path, including the first slash of the root partition:

 ./s3sync.rb -vrd /home/martin/test mrvanes-us:/home/martin/test

This does not yield an error, but neither a successfull backup when I check the upload in s3fox.
A 0 bytes sized empty-named directory is created, but is not accessible.

I'm fully aware that I may have handled against the rules of the readme, but I wanted to raise the point since it seems an obvious error to make (to me at least

On close inspection, it turns out s3cmd can read the backup (list shows the filenames), so it seems s3fox can not handle the first slash in the base prefix. Although this might be specific to s3fox I would suggest stripping any leading slash since it doesn't serve any purpose anyway. Or am I wrong?

Sorry, but I have to fail report on ALL attempts.
I too, use s3fox to administrate my buckets, but a EU bucket created in s3fox is not accessible with s3sync:

Code:

$ ./s3sync.rb -vrnd /home/martin/test mrvanes-s3fox-eu:pre
s3Prefix pre/test
localPrefix /home/martin/test
local node object init. Name: Path:/home/martin/test/ Size:38 Tag:d66759af42f282e1ba19144df2d405d0
s3TreeRecurse mrvanes-s3fox-eu pre/test
Creating new connection
Trying command list_bucket mrvanes-s3fox-eu max-keys 200 prefix pre/test delimiter / with 100 retries left
Response code: 403
S3 command failed:
list_bucket max-keys 200 prefix pre/test delimiter /
With result 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><RequestId>xxx</RequestId><SignatureProvided>xxx=</SignatureProvided><StringToSignBytes>47 45 54 0a 0a 0a 4d 6f 6e 2c 20 31 39 20 4e 6f 76 20 32 30 30 37 20 32 30 3a 33 35 3a 32 33 20 47 4d 54 0a 2f</StringToSignBytes><AWSAccessKeyId>xxx</AWSAccessKeyId><HostId>xxx</HostId><StringToSign>GET


Mon, 19 Nov 2007 20:35:23 GMT
/</StringToSign></Error>
S3 ERROR: #<Net::HTTPForbidden:0xb7c2d848>
./s3sync.rb:292:in `+': can't convert nil into Array (TypeError)
        from ./s3sync.rb:292:in `s3TreeRecurse'
        from ./s3sync.rb:348:in `main'
        from ./thread_generator.rb:79:in `call'
        from ./thread_generator.rb:79:in `initialize'
        from ./thread_generator.rb:76:in `new'
        from ./thread_generator.rb:76:in `initialize'
        from ./s3sync.rb:269:in `new'
        from ./s3sync.rb:269:in `main'
        from ./s3sync.rb:711

I get exactly the same errors on buckets created wich s3cmd btw, so that doesn't seem to help me.

On a sidenote, s3cmd doesn't seem to play nice with aws_calling_format, setting it to subdomain renders s3cmd useless, so I need to comment it whenever I use s3cmd on european buckets (create/list/delete).

So, so far, I haven't been able to successfully store anything on European s3.

I cleared my output from any secret looking info, but if it's helpfull to print everything and someone can convince me it's safe I'm more than willing to cooperate

Thx for the extremely quick fix!
I'll try it tonight but do I understand correctly this "feature" won't be transparent? It seems s3fox is able to handle it transparent, but it might of course keep track of the location internally?

Ok, so a quick glance learned you construct the complete path (bucket+key) outside generate_url. Too bad, this requires a lot more work now, since you need the bucket and the path separately when generating the URL for the above trick to work...

Just 5 minutes into trying s3sync I stumbled at exactly this problem and as far as I understand the fix is quite simple.
The only thing you need to do is change your URL scheme

from http://s3.amazonaws.com/<bucketname>
to http://<bucketname>.s3.amazonaws.com/

and DNS will do the rest... it's transparent, it's easy. As far as I understand.

I would change it myself if I knew ruby and where you construct your URL in the code.