s3cmd get use a lot or resource, 10 times more CPU the put

[danm]

Hi,

I notice the the get command use a lot of resource on my ec2.
Its 10 times more CPU the put. Is it a bug?

Here are some statistics:

/usr/bin/time  s3cmd put aaa:file /mnt/tmp/100m
1st try
0.59user 0.15system 1:09.99elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+4257minor)pagefaults 0swaps
2nd try
0.53user 0.20system 0:16.68elapsed 4%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+4254minor)pagefaults 0swaps

/usr/bin/time  s3cmd get aaa:file /mnt/tmp/100m
1st try
20.23user 1.77system 0:52.00elapsed 42%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+3366minor)pagefaults 0swaps
2nd try
10.19user 0.94system 0:26.86elapsed 41%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+3585minor)pagefaults 0swaps

Is it a bug?

[ferrix]

You'd have to set up profiling in ruby to see where the cpu is going.  I bet it's not in the s3sync code; the ruby http library is awful.

[danm]

Hi,

The problem is in s3cmd (not s3sync)

Code:

s3cmd get aaa:file /mnt/tmp/100m

As far as I understand it should do common http get request.  Some thing that shouldn't use huge CPU resources.

When using "wget" command instead of "s3cmd get"

Code:

/usr/bin/time wget http://mybucket.s3.amazonaws.com/100m

It much much faster:
0.00user 0.16system 0:08.87elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+478minor)pagefaults 0swaps

0.01user 0.11system 0:09.92elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+477minor)pagefaults 0swaps

Abut 0.16 of CPU second instead of 9 seconds!
Abut 10 second elapsed time instead of 30 seconds!

[ferrix]

Like I said, the http ruby lib is awful.  You might try to profile it if you want to see what's going on.  But my code does not do anything cpu intensive as far as I know.