S3Sync.net

Hi Greg Ferrix

Thank you for coming out with this excellent s3sync.rb.

However, I am more interested in s3cmd.rb, and need a 'create empty folder' feature.

I found something here:
http://developer.amazonwebservices.com/connect/thread.jspa?messageID=36210
...  could create an empty object for each directory and use a meta tag to flag it as directory ...

Wondering if you drop some direct hints on how to tweak your s3cmd.rb script to do this?

The reason why I want to create emtyp folder :

I noticed to that when I used s3cmb.rb to put something on Amazon, e.g,
s3cmd.rb  put  pa_svn:testdir2/testfile2b localtestdir002/localtestfile2b,
s3cmd.rb will create the folder testdir2 containing the file testfile2b on Amazon.

But when I use Firefox S3 organizer to delete the file testfile2b on Amazon,
the testdir2 on Amazon also disappeared.
It seems that the folder testdir2 created by s3cmd.rb cannot be empty.
This is unlike in Firefox S3 organizer where you can create an empty folder.

When I do this,
s3cmd.rb  put  pa_svn:testdir4  localtestdir4  ,
s3cmd.rb will create something that looks like a file on Amazon (Firefox S3 organizer sees a file 'testdir4').
Then I do this,
s3cmd.rb  put  pa_svn:testdir4/testfile4  localtestdir4/localtestfile4,
Firefox S3 organizer still sees the same thing, something that looks like a file 'testdir4'
and when I download using Firefox S3 organizer, 'testdir4' is not a directory but a file which is unreadable.

I guess s3cmd.rb is not fully compatible with Firefox S3 organizer.
But if there is a command in s3cmd.rb to create an empty folder on Amazon, I think
if I create the folder first on Amazon before putting the file there, it will
be compatible with Firefox S3 organizer.

Any direct hints on how to create an empty folder on Amazon S3?

Thank you very much.

Short answer: sorry, no.

As I mentioned before I'm not going to turn s3cmd into s3sync.  They are for fundamentally different things.  You still don't understand that S3 does not have a concept files and directories.  It just has nodes.  The concept of files and directories is added differently by each kind of tool.

s3sync makes directory nodes in exactly the way mentioned from the thread you linked.. It is a node containing meta data flagging it as a directory.  And those are the things that your 'firefox s3 organizer' are seeing as files.  So with s3sync I have already done this thing that you think you want.

I don't know about firefox s3 organizer or how (or if) it has some node format that it would interpret as an empty directory.  I have very good design reasons for doing directories the way I've done them and so it's not likely to change. 

As I have stated before, the only compatibility goal of s3sync is to make web browser operations on public nodes work as transparently as possible.  Now: If someone else wants to make an s3 shell that is interoperable with "firefox s3 explorer" and contains more types of operations and features that you would like, then they are welcome to do so.  They can even start by using my code if it helps.. it's a free license.  But I'm not going to build that tool myself because I don't have any use for it. 

I just ran into the "problem" of the S3 Firefox Organizer not recognizing folders created by s3sync today. The ff extension designates folders by appending '_$folder$' to the node name. My shameful hack to get these two tools to live together was to add a check for nodes that are 38 bytes long(the length of the s3sync meta data) and treat these as folders. I would like to just check the contents, but that would require reading each node to determine if it is a directory. It's not a perfect solution, but it works for me.

Is not having a standard way of identifying folders going to be a problem when more and more tools come out for managing S3? It appears that way to me.

Just check the etag.. the contents of the folders are always the same, so they will md5 to the same etag.  That's how *I* check what ones are folders.  I do that because the etag comes down in the item list and I don't have to head or get each node to check what it is.

And it goes without saying that tacking on some crap to the name is not a good solution.

As for cross compatibility, I can't say that I care really.  But the code is out there, so anyone can modify it, and post it up on the wiki.  In fact if it's a classy enough mod (with an option switch or something) I could even roll it in to the source.

But once again, I'm not changing the way it works by default, because my way is best.  Nyah.  :P

the way I do it is mark directories and symlinks in the metadata, then when I do the HEAD call I know I can restore them without doing a GET, just by using the metadata content.  as previously mentioned, I punt on hardlinks (I PUT them once for each link, i.e. break the links).  I  do the HEAD calls first to check the etag so as not to retrieve something I already have.  my typical file is around 2-3 MB, so the overhead of the HEAD is in the noise.

Etags come down in a list.. why would you have to head the items too?