|
Title: s3cmd get use a lot or resource, 10 times more CPU the put Post by: danm on May 10, 2008, 01:53:47 PM Hi,
I notice the the get command use a lot of resource on my ec2. Its 10 times more CPU the put. Is it a bug? Here are some statistics: /usr/bin/time s3cmd put aaa:file /mnt/tmp/100m 1st try 0.59user 0.15system 1:09.99elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+4257minor)pagefaults 0swaps 2nd try 0.53user 0.20system 0:16.68elapsed 4%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+4254minor)pagefaults 0swaps /usr/bin/time s3cmd get aaa:file /mnt/tmp/100m 1st try 20.23user 1.77system 0:52.00elapsed 42%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+3366minor)pagefaults 0swaps 2nd try 10.19user 0.94system 0:26.86elapsed 41%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+3585minor)pagefaults 0swaps Is it a bug? Title: Re: s3cmd get use a lot or resource, 10 times more CPU the put Post by: ferrix on May 10, 2008, 08:16:07 PM You'd have to set up profiling in ruby to see where the cpu is going. I bet it's not in the s3sync code; the ruby http library is awful.
Title: Re: s3cmd get use a lot or resource, 10 times more CPU the put Post by: danm on May 11, 2008, 12:17:32 AM Hi,
The problem is in s3cmd (not s3sync) Code: s3cmd get aaa:file /mnt/tmp/100m As far as I understand it should do common http get request. Some thing that shouldn't use huge CPU resources. When using "wget" command instead of "s3cmd get" Code: /usr/bin/time wget http://mybucket.s3.amazonaws.com/100m It much much faster: 0.00user 0.16system 0:08.87elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+478minor)pagefaults 0swaps 0.01user 0.11system 0:09.92elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+477minor)pagefaults 0swaps Abut 0.16 of CPU second instead of 9 seconds! Abut 10 second elapsed time instead of 30 seconds! Title: Re: s3cmd get use a lot or resource, 10 times more CPU the put Post by: ferrix on May 11, 2008, 02:37:42 AM Like I said, the http ruby lib is awful. You might try to profile it if you want to see what's going on. But my code does not do anything cpu intensive as far as I know.
|