S3Sync.net

(I think this is a bug, but not completely sure.)

So I'm trying to get data off s3. The data looks something like:
/data/foo/foo/bar.txt
/data/foo/foo/baz.txt

the directory:
/data/foo/

contains only subdirectories; no files.

when I try to do:
ruby s3sync.rb -r --delete mybucket:/data/ /home/me/data-test

I get:
Could not mkdir /home/me/data-test/foo/foo: No such file or directory - /home/me/data-test/foo/foo

/home/me/data-test/ exists; I think it is failing because data-test/foo/ doesn't exist, and it is trying to create data-test/foo/foo/ without creating data-test/foo/ first.

Is this a bug? Silliness on my end? I'm pretty sure bug, but maybe there is a subtlety/design decision I'm missing here.

Forgot to add that if I create the local directory /home/me/data-test/foo/  that it works fine. But if I have to manually create every directory which has no files in it, that isn't going to scale very well :/

Did you *upload* the files using s3sync.rb?  If so, then directory nodes should have been placed which would, upon retreival, do all the correct local dir creation.

I had the same problem of syncing data from s3 to a local filesystem where the data *never* was before: The missing directories are simply not created on the fly.

To fix this insert the following in the require section:

require 'ftools'

And add File.makedirs() as follows:

def updateFrom(fromNode)
    if fromNode.respond_to?(:to_stream)
        fName = @path + '.s3syncTemp'
        File.makedirs(File.dirname(@path)) # This is the important part!

    [...]
end

Perhapes, this could make it to the next release, please?

Best regards, Jan

Not promiscuously creating directories locally when there are no dir nodes stored @S3 for them is a design decision.  Dow would it know what permissions and owner to use, etc.