S3Sync.net
February 02, 2014, 01:20:11 PM *
Welcome, Guest. Please login or register.

Login with username, password and session length
 
   Home   Help Search Login Register  
Pages: [1]
  Print  
Author Topic: show more than 1000 files with s3cmd.rb  (Read 8028 times)
maelcum
Newbie
*
Posts: 43



View Profile
« on: April 26, 2007, 12:12:24 PM »

Hi,

how can I list everything I have stored in a bucket with s3cmd WITHOUT having to repeatedly answer the "more (Y/n)" question?

I'd like to pipe s3cmd's output to sed or awk (or any other util) but unfortunately, the maximum number of line for a listing is 1000, and after that, I'd have to answer the pagination question.

Entering anything beyond "1000" gives the following error:
Code:
  S3 command failed:
  list_bucket NASa max-keys 1001 prefix   
  With result 400 Bad Request
  ./s3cmd.rb:122:in `s3cmdMain': undefined method `is_truncated' for nil:NilClass (NoMethodError)
          from ./s3cmd.rb:207

Could this be some limitation on S3's side? With JetS3t Cockpit I do see everything, but then, they are fetching this in increments of 1000, I assume...

Any idea?
Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #1 on: April 27, 2007, 04:27:48 PM »

I just need to get around to making an option for that.

Yes, 1000-per is a limitation of S3, but there's no reason why I can't code the tool to work with that and still display a unified output.
Logged
maelcum
Newbie
*
Posts: 43



View Profile
« Reply #2 on: June 05, 2007, 04:52:24 PM »

Hi,

just wanted to add that this request could probably be closed.
If I pipe a yes to the "s3cmd.rb list <somebucket:>" then I still get the "More Y/n: " string in my output, but that can easily be cleaned with a simple "sed 's/More Y\/n://'".

Kinda workaround, and two more apps needed to work, but it turns out okay.

Cheers.
Logged
jandrea
Newbie
*
Posts: 3


View Profile
« Reply #3 on: May 28, 2008, 02:28:42 PM »

I'm looking for further details.

I see that the option of max/page exists. If I set this to 123456789 then I don't get any paging over my 450 objects.
   But are you saying that if I have more than 1000 objects then even using that option I only ever see 1000 objects ? Will need to test this.

So new feature requests is:
Paging of output is not the Unix way (see original comment on piping to awk/sed,etc.). There should be no paging. To enable paging the option of max/page should be used, but it should default to zero which means infinity = therefor no paging.

Another new feature request:
There should be no output of a line of dashes at the beginning output. Again the Unix way is to provide none of this extranious output.
Logged
jandrea
Newbie
*
Posts: 3


View Profile
« Reply #4 on: May 28, 2008, 04:00:43 PM »

Test results.

Created a bucket with 1019 objects (number.dat as key name)

Tried: s3cmd.rb list bucket.test 12345
Result: It prompted when it got to object # 1000. Looks like option max/page > 1000 has no use.
version 1.2.5

Tried other product 'aws' (perl + curl):
Result: it stopped at # 1000.
version 1.10

Tried other product 's3cmd' (Python)
Result: it stopped at # 1000.
version 0.9.4
Logged
D.Grere
Newbie
*
Posts: 1


View Profile WWW
« Reply #5 on: August 27, 2009, 03:46:26 AM »

Or, make it an option to do so. I check over a dozen accounts, and more often than not, the balloon tip is filled with the accounts that show zero messages, while the ones with messages are truncated. Thanks Cool feature though
Logged
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2013, Simple Machines Valid XHTML 1.0! Valid CSS!