S3sync seems to be uploading/checking but s3Fox doesn't show the files

[tnik]

Strange problem here. I have just recently moved thousands of files to our backup server, and I have our backup server using s3sync to amazon. Log file shows that the files were created (eg. Create node A-E/ACME_DYNAMICS/ACM002C.M09) but when I check from s3fox, the files aren't there.

this is the script I'm running..

Code:

#!/bin/bash

BUCKETPATH=backup-data:/toolcrib/
BACKUPPATH=/backup_data/toolcrib/
EXCLUDE=

date
echo start toolcrib bu...
cd /root/s3sync && /usr/bin/ruby s3sync.rb -rvs ${BACKUPPATH} ${BUCKETPATH}
echo done toolcrib bu...
date

If I were to run it right now in debug mode, here is an clip from debug output.

Code:

local item /backup_data/toolcrib/A-E/ALABAMA_METALS/24-135 WEAR TRACK SPLICE JOINTS.3d
local node object init. Name:A-E/ALABAMA_METALS/24-135 WEAR TRACK SPLICE JOINTS.3d Path:/backup_data/toolcrib/A-E/ALABAMA_METALS/24-135 WEAR TRACK SPLICE JOINTS.3d Size:20176 Tag:4495f29583f73507fc0897f68fb33bce
prefix found: /A-E/ALABAMA_METALS/
s3TreeRecurse backup-data /toolcrib /A-E/ALABAMA_METALS/
Trying command list_bucket backup-data max-keys 200 prefix /toolcrib/A-E/ALABAMA_METALS/ delimiter / with 100 retries left
Response code: 200
S3 item /toolcrib/A-E/ALABAMA_METALS/24-135 WEAR TRACK SPLICE JOINTS.3d
s3 node object init. Name:A-E/ALABAMA_METALS/24-135 WEAR TRACK SPLICE JOINTS.3d Path:/toolcrib/A-E/ALABAMA_METALS/24-135 WEAR TRACK SPLICE JOINTS.3d Size:20176 Tag:4495f29583f73507fc0897f68fb33bce
source: A-E/ALABAMA_METALS/24-135 WEAR TRACK SPLICE JOINTS.3d
dest: A-E/ALABAMA_METALS/24-135 WEAR TRACK SPLICE JOINTS.3d
Node A-E/ALABAMA_METALS/24-135 WEAR TRACK SPLICE JOINTS.3d unchanged

Going to my backup server in the 'ALABAMA_METALS' directory, there are 125 files, s3sync shows that those 125 files have been uploaded and unchanged from previous syncs, but s3fox doesn't show anything being in there 

any ideas?

[tnik]

ok.. for sh_ts and giggles, I went ahead and tried to do a restore with s3sync, and it created all the files that 's3fox' doesn't show.. verrrry strange..  is there another windows based program that I could use to 'double' check s3, or any info that I should know why s3fox is blind to some files but not others?

[tnik]

sucks talking to myself, but at least I'm updating

ok, files seem to be transfering fine.. strange that I can't see them in any other client other than using s3cmd list. is there a way to get the listing of the ACL of the files from s3cmd? maybe somehow some files have different permissions setup?

I was really hoping I would be able to use s3fox or the like to just grab a file here or there when needed, and not have to goto the server and s3cmd when needed.

[ferrix]

Already loads of threads talking about s3fox incompat with s3sync, and about the fiction of "directories" on s3.  I have mixed luck with fox.  Sometimes it shows a "dir" as a file, but if you add that to the path manually it will go "in" to the "dir" that it didn't think was there.

[tnik]

oh I'm sorry, I did search around a bit.. guess I should have dove deeper

yea it's strange, even the 'manual' approach doesn't work when going into the directories, which I guess is just a placeholder..

I did find a win based program, Bucket-Explorer, that you can switch between viewing in 'folder view' or full path view.. doing full path view I can see the mysterious keys/files (sometimes only by entering the absolute path), but as for being in 'folder view' even entering the path manually doesn't jive.

and if there is anything I can do to try to finger the culprit on this, just yell..