Show Upload Time in s3cmd's list command

[cm_gui]

Hi Greg

Thank you for the excellent s3sync script.

Is it possible to have the s3cmd's list command show the Upload Time of the files?

This is the reason why we need this feature:
Because of Amazon 5GB limit on object size (2GB actually due to an Amazon bug), we cannot use s3sync.rb.
The data folders we want to s3sync are bigger than 5GB and we cannot split them up into smaller parts.

So I am using s3cmd's put command to put our files one by one onto Amazon S3 every night; putting only files that were modified within the last 24 hours (i.e. since the time of the last backup) --- however this nightly script will cause problem if it doesn't run for a day or two
--  in which case, the data during these days would not be backed up to Amazon.

It would be better if the s3cmd's list command can show the Upload Time of the files -- so that we can compare the Upload Time of the files on Amazon with the Modified date of the files on our server, and then put the newer files onto Amazon.   

Thank you very much.

[lowflyinghawk]

s3sync puts the files in a folder one by one already right?  so if you can s3cmd put multiple individual files, then can't you use s3sync on the folders?

if you do a HEAD before PUT it is possible to check a file for changes and not do the PUT at all if it's the same, i.e. approximately the way rsync behaves.  if s3sync does this already (and I assume it does), then cm_gui could just put the whole folder and only the changed files will actually be transferred.  this is the whole idea isn't it?

[cm_gui]

No, I cannot use s3sync.rb to put my folders to Amazon because my folders are  much bigger than 5GB, and I cannot divide my folders into smaller parts to s3sync.

I have to use s3cmd.rb's put to put the files one by one.
I want to be able to read the upload time of the files already on Amazon, and then upload newer files from my server.

Is there a HEAD command in s3sync or s3cmd?
How do I use it?  Thanks.

s3sync puts the files in a folder one by one already right?  so if you can s3cmd put multiple individual files, then can't you use s3sync on the folders?

if you do a HEAD before PUT it is possible to check a file for changes and not do the PUT at all if it's the same, i.e. approximately the way rsync behaves.  if s3sync does this already (and I assume it does), then cm_gui could just put the whole folder and only the changed files will actually be transferred.  this is the whole idea isn't it?

[lowflyinghawk]

you are mistaken, the size of your folders is irrelevant, only the size of the individual files matters. see your other thread.

the HEAD comment was intended for the developer, not for you.

[ferrix]

cm_gui: I'm sorry you're having trouble, but posting the same request in another way without understanding the first response isn't going to get us anywhere.

As we have repeatedly said, the object size limit, as well as the 2GB bug, is for an individual S3 node (in this case, files), not collections of them (folders).  s3sync does make a node representing the folder, but it is simply a small placeholder to record the permissions etc.  It will in no way impact size limitations.

[cm_gui]

Hi Greg

Thank you for the reply.  I get it now.

One question -- are s3cmd and s3sync compatible?

I've posted this question in  http://s3sync.net/forum/index.php?topic=18.msg84#msg84

When I upload a folder, say /data/folder1, to Amazon using s3sync.rb,
I cannot use s3cmd's get command to download subfolders or individual files inside /data/folder1 from Amazon.

When I upload individuals files one by one to Amazon using s3cmd's put command -- and duplicating
the same directory structure /data/folder1 on Amazon,  I cannot use s3sync to download anything
from the /data/folder1 at all.

[ferrix]

Oops I missed that.  I'll answer it over there.

[cm_gui]

I managed to get s3cmd.rb to print the file's last_modified (upload time) by adding the
following line after line no. 118 in s3cmd.rb.
puts Time.parse( item.last_modified ).to_i