Ah... I think I've found the issue...

It seems thats if I include an S3 prefix, e.g.

$ ruby s3sync/s3sync.rb MY_BACKUP_ROOT/ bucketname:test -v

then every file gets uploaded every time. On the other hand, if I remove the prefix, e.g.

$ ruby s3sync/s3sync.rb MY_BACKUP_ROOT/ bucketname: -v

then only the missing files get uploaded.

Thanks!

I'd be happy to... although it doesn't seem to be bucket specific; I've created several buckets and seen the same behavior on each. This is probably a dumb question, but how do I use s3cmd to set ACLs? I thought I could do something like

s3cmd.rb createbucket martinsbucket x-amz-acl:public-read-write

But this fails with a 400 bad request.

Thanks for your help.

Sure. I deleted the files off s3 (using s3cmd), and then re-ran the same command with -d twice. Here is the output. Thanks.

martin@dev:~/dev-server-backup$ s3sync/s3sync.rb test/ bucketname:test -v -d
s3Prefix test
localPrefix /home/martin/dev-server-backup/test/
localTreeRecurse /home/martin/dev-server-backup/test
Test /home/martin/dev-server-backup/test/test2
Test /home/martin/dev-server-backup/test/test1
Test /home/martin/dev-server-backup/test/test4
Test /home/martin/dev-server-backup/test/test5
Test /home/martin/dev-server-backup/test/test3
local item /home/martin/dev-server-backup/test/test1
local node object init. Name:test1 Path:/home/martin/dev-server-backup/test/test1 Size:6 Tag:3e7705498e8be60520841409ebc69bc1
s3TreeRecurse bucketname test
Creating new connection
Trying command list_bucket bucketname max-keys 200 prefix test delimiter / with 100 retries left
Response code: 200
prefix found: /
source: test1
s3 node object init. Name:test1 Path:test/test1 Size: Tag:
Create node test1
test/test1
File extension: test/test1
Trying command put bucketname test/test1 #<S3::S3Object:0xb7a370d4> Content-Length 6 with 100 retries left
Response code: 200
local item /home/martin/dev-server-backup/test/test2
martin@dev:~/dev-server-backup$ s3sync/s3sync.rb test/ bucketname:test -v -d
s3Prefix test
localPrefix /home/martin/dev-server-backup/test/
localTreeRecurse /home/martin/dev-server-backup/test
Test /home/martin/dev-server-backup/test/test2
Test /home/martin/dev-server-backup/test/test1
Test /home/martin/dev-server-backup/test/test4
Test /home/martin/dev-server-backup/test/test5
Test /home/martin/dev-server-backup/test/test3
local item /home/martin/dev-server-backup/test/test1
local node object init. Name:test1 Path:/home/martin/dev-server-backup/test/test1 Size:6 Tag:3e7705498e8be60520841409ebc69bc1
s3TreeRecurse bucketname test
Creating new connection
Trying command list_bucket bucketname max-keys 200 prefix test delimiter / with 100 retries left
Response code: 200
source: test1
s3 node object init. Name:test1 Path:test/test1 Size: Tag:
Create node test1
test/test1
File extension: test/test1
Trying command put bucketname test/test1 #<S3::S3Object:0xb7964170> Content-Length 6 with 100 retries left
Response code: 200
local item /home/martin/dev-server-backup/test/test2
local node object init. Name:test2 Path:/home/martin/dev-server-backup/test/test2 Size:6 Tag:126a8a51b9d1bbd07fddc65819a542c3
source: test2
s3 node object init. Name:test2 Path:test/test2 Size: Tag:
Create node test2
test/test2
File extension: test/test2
Trying command put bucketname test/test2 #<S3::S3Object:0xb7958370> Content-Length 6 with 100 retries left
Response code: 200
local item /home/martin/dev-server-backup/test/test3
local node object init. Name:test3 Path:/home/martin/dev-server-backup/test/test3 Size:6 Tag:3bc3be114fb6323adc5b0ad7422d193a
source: test3
s3 node object init. Name:test3 Path:test/test3 Size: Tag:
Create node test3
test/test3
File extension: test/test3
Trying command put bucketname test/test3 #<S3::S3Object:0xb794c4a8> Content-Length 6 with 100 retries left
Response code: 200
local item /home/martin/dev-server-backup/test/test4
local node object init. Name:test4 Path:/home/martin/dev-server-backup/test/test4 Size:6 Tag:b5163cf270a3fbac34827c4a2713eef4
source: test4
s3 node object init. Name:test4 Path:test/test4 Size: Tag:
Create node test4
test/test4
File extension: test/test4
Trying command put bucketname test/test4 #<S3::S3Object:0xb793f654> Content-Length 6 with 100 retries left
Response code: 200
local item /home/martin/dev-server-backup/test/test5
local node object init. Name:test5 Path:/home/martin/dev-server-backup/test/test5 Size:6 Tag:bb4da129079c12d4ddaee64ba79a03ff
source: test5
s3 node object init. Name:test5 Path:test/test5 Size: Tag:
Create node test5
test/test5
File extension: test/test5
Trying command put bucketname test/test5 #<S3::S3Object:0xb79361f8> Content-Length 6 with 100 retries left
Response code: 200
martin@dev:~/dev-server-backup$ s3sync/s3sync.rb test/ bucketname:test -v -d
s3Prefix test
localPrefix /home/martin/dev-server-backup/test/
localTreeRecurse /home/martin/dev-server-backup/test
Test /home/martin/dev-server-backup/test/test2
Test /home/martin/dev-server-backup/test/test1
Test /home/martin/dev-server-backup/test/test4
Test /home/martin/dev-server-backup/test/test5
Test /home/martin/dev-server-backup/test/test3
local item /home/martin/dev-server-backup/test/test1
local node object init. Name:test1 Path:/home/martin/dev-server-backup/test/test1 Size:6 Tag:3e7705498e8be60520841409ebc69bc1
s3TreeRecurse bucketname test
Creating new connection
Trying command list_bucket bucketname max-keys 200 prefix test delimiter / with 100 retries left
Response code: 200
prefix found: /
source: test1
s3 node object init. Name:test1 Path:test/test1 Size: Tag:
Create node test1
test/test1
File extension: test/test1
Trying command put bucketname test/test1 #<S3::S3Object:0xb79cf150> Content-Length 6 with 100 retries left
Response code: 200
local item /home/martin/dev-server-backup/test/test2
local node object init. Name:test2 Path:/home/martin/dev-server-backup/test/test2 Size:6 Tag:126a8a51b9d1bbd07fddc65819a542c3
source: test2
s3 node object init. Name:test2 Path:test/test2 Size: Tag:
Create node test2
test/test2
File extension: test/test2
Trying command put bucketname test/test2 #<S3::S3Object:0xb79c3198> Content-Length 6 with 100 retries left
Response code: 200
local item /home/martin/dev-server-backup/test/test3
local node object init. Name:test3 Path:/home/martin/dev-server-backup/test/test3 Size:6 Tag:3bc3be114fb6323adc5b0ad7422d193a
source: test3
s3 node object init. Name:test3 Path:test/test3 Size: Tag:
Create node test3
test/test3
File extension: test/test3
Trying command put bucketname test/test3 #<S3::S3Object:0xb79b72bc> Content-Length 6 with 100 retries left
Response code: 200
local item /home/martin/dev-server-backup/test/test4
local node object init. Name:test4 Path:/home/martin/dev-server-backup/test/test4 Size:6 Tag:b5163cf270a3fbac34827c4a2713eef4
source: test4
s3 node object init. Name:test4 Path:test/test4 Size: Tag:
Create node test4
test/test4
File extension: test/test4
Trying command put bucketname test/test4 #<S3::S3Object:0xb79aa418> Content-Length 6 with 100 retries left
Response code: 200
local item /home/martin/dev-server-backup/test/test5
local node object init. Name:test5 Path:/home/martin/dev-server-backup/test/test5 Size:6 Tag:bb4da129079c12d4ddaee64ba79a03ff
source: test5
s3 node object init. Name:test5 Path:test/test5 Size: Tag:
Create node test5
test/test5
File extension: test/test5
Trying command put bucketname test/test5 #<S3::S3Object:0xb79a1070> Content-Length 6 with 100 retries left
Response code: 200
martin@dev:~/dev-server-backup$

Hi guys,

I'm trying to figure out what, if anything, I'm missing here. It was my understanding that, like rsync, s3sync is supposed to not recopy files that already exist. It seems to be suggested that this is the case in a forum thread I found (http://developer.amazonwebservices.com/connect/thread.jspa?threadID=11975&start=15&tstart=0). But this doesn't seem to be actually happening, as if I resync the same directory repeatedly, the same files get copied over and over:

martin@dev:~/dev-server-backup$ s3sync/s3sync.rb test/ bucketname:test -v
Create node test1
Create node test2
Create node test3
Create node test4
Create node test5
martin@dev:~/dev-server-backup$ s3sync/s3sync.rb test/ bucketname:test -v
Create node test1
Create node test2
Create node test3
Create node test4
Create node test5

Any insight would be appreciated.

Thanks,
Martin