Bandwidth Throttling

[timdenike]

This may be a stretch, but a bandwidth throttling option would be pretty handy.  I'm experimenting with using s3sync for syncing encrypted backups to S3, and the daily sync is likely to knock our bandwidth commit up a couple notches. 

Tim Denike

[AltJ]

+1  I was backing up a colocated box w/s3sync until the datacenter informed me of my bandwidth usage (I'm billed on 95th percentile.)  I don't care if the backup takes a couple of hours to complete, there are many hours during the night when the system is pretty much idle.

[tienshiao]

I had a similar need. I run s3sync over my DSL connection, but I could only leave it running at certain times because if I max out my upstream, everything comes to a crawl.

I've added support for throttling upstream and downstream.

--ulimit=<bytes/s>
--dlimit=<bytes/s>

I primarily use it for upload throttling and it seems to work well enough. I did a quick test for download throttling and it seems to work also.


How it works:
It depends on progress stream (since it need to calculate the rate), so we always use progress stream now. To toggle displaying of progress, I changed that to be a parameter in the constructor.

Rate is calculated whenever read() or write() is called. If the rate is greater than the limit, it calculates how long it would need to sleep for to bring it down to the limit, and then it sleeps that amount.


(Un)fortunately the patch is intermingled with my Content-MD5 patch.


Tienshiao

[Scott]

Try trickle on linux (unix??) http://www.linux.com/archive/articles/61293?tid=100&tid=47&tid=13

It lets you throttle a TCP application, kind of like nice.  I haven't tried it yet, the docs say the limits are in KB not Kb.  So to keep it at 200kbps, I'm gonna try 50KBps to send 15g in 2 days or so.

trickle -u 50 -d 50 s3sync.rb ....

[After some minimal testing, the numbers coming out of ntop do show it works but you'll have to tweak the trickle inputs to get the desired output.   On my system -d 20 -u 20 is reported as roughly 200Kbps in ntop.  50/50 was reported as 500Kbps]

[morgant]

I'd like this feature as well.