S3Sync.net
February 02, 2014, 01:30:15 PM *
Welcome, Guest. Please login or register.

Login with username, password and session length
 
   Home   Help Search Login Register  
Pages: [1]
  Print  
Author Topic: s3cmd get use a lot or resource, 10 times more CPU the put  (Read 4533 times)
danm
Newbie
*
Posts: 2


View Profile
« on: May 10, 2008, 01:53:47 PM »

Hi,

I notice the the get command use a lot of resource on my ec2.
Its 10 times more CPU the put. Is it a bug?

Here are some statistics:

/usr/bin/time  s3cmd put aaa:file /mnt/tmp/100m
1st try
0.59user 0.15system 1:09.99elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+4257minor)pagefaults 0swaps
2nd try
0.53user 0.20system 0:16.68elapsed 4%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+4254minor)pagefaults 0swaps

/usr/bin/time  s3cmd get aaa:file /mnt/tmp/100m
1st try
20.23user 1.77system 0:52.00elapsed 42%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+3366minor)pagefaults 0swaps
2nd try
10.19user 0.94system 0:26.86elapsed 41%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+3585minor)pagefaults 0swaps

Is it a bug?
Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #1 on: May 10, 2008, 08:16:07 PM »

You'd have to set up profiling in ruby to see where the cpu is going.  I bet it's not in the s3sync code; the ruby http library is awful.
Logged
danm
Newbie
*
Posts: 2


View Profile
« Reply #2 on: May 11, 2008, 12:17:32 AM »

Hi,

The problem is in s3cmd (not s3sync)

Code:
s3cmd get aaa:file /mnt/tmp/100m

As far as I understand it should do common http get request.  Some thing that shouldn't use huge CPU resources.

When using "wget" command instead of "s3cmd get"
Code:
/usr/bin/time wget http://mybucket.s3.amazonaws.com/100m

It much much faster:
0.00user 0.16system 0:08.87elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+478minor)pagefaults 0swaps

0.01user 0.11system 0:09.92elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+477minor)pagefaults 0swaps

Abut 0.16 of CPU second instead of 9 seconds!
Abut 10 second elapsed time instead of 30 seconds!

« Last Edit: May 11, 2008, 12:41:43 AM by danm » Logged
ferrix
Sr. Member
****
Posts: 363


(I am greg13070 on AWS forum)


View Profile
« Reply #3 on: May 11, 2008, 02:37:42 AM »

Like I said, the http ruby lib is awful.  You might try to profile it if you want to see what's going on.  But my code does not do anything cpu intensive as far as I know.
Logged
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2013, Simple Machines Valid XHTML 1.0! Valid CSS!