|
Title: show more than 1000 files with s3cmd.rb Post by: maelcum on April 26, 2007, 12:12:24 PM Hi,
how can I list everything I have stored in a bucket with s3cmd WITHOUT having to repeatedly answer the "more (Y/n)" question? I'd like to pipe s3cmd's output to sed or awk (or any other util) but unfortunately, the maximum number of line for a listing is 1000, and after that, I'd have to answer the pagination question. Entering anything beyond "1000" gives the following error: Code: S3 command failed: list_bucket NASa max-keys 1001 prefix With result 400 Bad Request ./s3cmd.rb:122:in `s3cmdMain': undefined method `is_truncated' for nil:NilClass (NoMethodError) from ./s3cmd.rb:207 Could this be some limitation on S3's side? With JetS3t Cockpit I do see everything, but then, they are fetching this in increments of 1000, I assume... Any idea? Title: Re: show more than 1000 files with s3cmd.rb Post by: ferrix on April 27, 2007, 04:27:48 PM I just need to get around to making an option for that.
Yes, 1000-per is a limitation of S3, but there's no reason why I can't code the tool to work with that and still display a unified output. Title: Re: show more than 1000 files with s3cmd.rb Post by: maelcum on June 05, 2007, 04:52:24 PM Hi,
just wanted to add that this request could probably be closed. If I pipe a yes to the "s3cmd.rb list <somebucket:>" then I still get the "More Y/n: " string in my output, but that can easily be cleaned with a simple "sed 's/More Y\/n://'". Kinda workaround, and two more apps needed to work, but it turns out okay. Cheers. Title: Re: show more than 1000 files with s3cmd.rb Post by: jandrea on May 28, 2008, 02:28:42 PM I'm looking for further details.
I see that the option of max/page exists. If I set this to 123456789 then I don't get any paging over my 450 objects. But are you saying that if I have more than 1000 objects then even using that option I only ever see 1000 objects ? Will need to test this. So new feature requests is: Paging of output is not the Unix way (see original comment on piping to awk/sed,etc.). There should be no paging. To enable paging the option of max/page should be used, but it should default to zero which means infinity = therefor no paging. Another new feature request: There should be no output of a line of dashes at the beginning output. Again the Unix way is to provide none of this extranious output. Title: Re: show more than 1000 files with s3cmd.rb Post by: jandrea on May 28, 2008, 04:00:43 PM Test results.
Created a bucket with 1019 objects (number.dat as key name) Tried: s3cmd.rb list bucket.test 12345 Result: It prompted when it got to object # 1000. Looks like option max/page > 1000 has no use. version 1.2.5 Tried other product 'aws' (perl + curl): Result: it stopped at # 1000. version 1.10 Tried other product 's3cmd' (Python) Result: it stopped at # 1000. version 0.9.4 Title: show more than 1000 files with s3cmd rb Post by: D.Grere on August 27, 2009, 03:46:26 AM Or, make it an option to do so. I check over a dozen accounts, and more often than not, the balloon tip is filled with the accounts that show zero messages, while the ones with messages are truncated. Thanks Cool feature though
|