show more than 1000 files with s3cmd.rb

[maelcum]

Hi,

how can I list everything I have stored in a bucket with s3cmd WITHOUT having to repeatedly answer the "more (Y/n)" question?

I'd like to pipe s3cmd's output to sed or awk (or any other util) but unfortunately, the maximum number of line for a listing is 1000, and after that, I'd have to answer the pagination question.

Entering anything beyond "1000" gives the following error:

Code:

  S3 command failed:
  list_bucket NASa max-keys 1001 prefix   
  With result 400 Bad Request
  ./s3cmd.rb:122:in `s3cmdMain': undefined method `is_truncated' for nil:NilClass (NoMethodError)
          from ./s3cmd.rb:207

Could this be some limitation on S3's side? With JetS3t Cockpit I do see everything, but then, they are fetching this in increments of 1000, I assume...

Any idea?

[ferrix]

I just need to get around to making an option for that.

Yes, 1000-per is a limitation of S3, but there's no reason why I can't code the tool to work with that and still display a unified output.

[maelcum]

Hi,

just wanted to add that this request could probably be closed.
If I pipe a yes to the "s3cmd.rb list <somebucket:>" then I still get the "More Y/n: " string in my output, but that can easily be cleaned with a simple "sed 's/More Y\/n://'".

Kinda workaround, and two more apps needed to work, but it turns out okay.

Cheers.

[jandrea]

I'm looking for further details.

I see that the option of max/page exists. If I set this to 123456789 then I don't get any paging over my 450 objects.
   But are you saying that if I have more than 1000 objects then even using that option I only ever see 1000 objects ? Will need to test this.

So new feature requests is:
Paging of output is not the Unix way (see original comment on piping to awk/sed,etc.). There should be no paging. To enable paging the option of max/page should be used, but it should default to zero which means infinity = therefor no paging.

Another new feature request:
There should be no output of a line of dashes at the beginning output. Again the Unix way is to provide none of this extranious output.

[jandrea]

Test results.

Created a bucket with 1019 objects (number.dat as key name)

Tried: s3cmd.rb list bucket.test 12345
Result: It prompted when it got to object # 1000. Looks like option max/page > 1000 has no use.
version 1.2.5

Tried other product 'aws' (perl + curl):
Result: it stopped at # 1000.
version 1.10

Tried other product 's3cmd' (Python)
Result: it stopped at # 1000.
version 0.9.4

[D.Grere]

Or, make it an option to do so. I check over a dozen accounts, and more often than not, the balloon tip is filled with the accounts that show zero messages, while the ones with messages are truncated. Thanks Cool feature though