S3Sync.net
February 02, 2014, 01:27:21 PM *
Welcome, Guest. Please login or register.

Login with username, password and session length
 
   Home   Help Search Login Register  
Pages: [1]
  Print  
Author Topic: Bandwidth Throttling  (Read 8536 times)
timdenike
Newbie
*
Posts: 1


View Profile
« on: November 13, 2007, 06:05:10 PM »

This may be a stretch, but a bandwidth throttling option would be pretty handy.  I'm experimenting with using s3sync for syncing encrypted backups to S3, and the daily sync is likely to knock our bandwidth commit up a couple notches. 

Tim Denike
Logged
AltJ
Newbie
*
Posts: 2


View Profile
« Reply #1 on: May 29, 2008, 06:30:13 PM »

+1  I was backing up a colocated box w/s3sync until the datacenter informed me of my bandwidth usage (I'm billed on 95th percentile.)  I don't care if the backup takes a couple of hours to complete, there are many hours during the night when the system is pretty much idle.
Logged
tienshiao
Newbie
*
Posts: 3


View Profile WWW
« Reply #2 on: July 24, 2008, 11:07:39 PM »

I had a similar need. I run s3sync over my DSL connection, but I could only leave it running at certain times because if I max out my upstream, everything comes to a crawl.

I've added support for throttling upstream and downstream.

--ulimit=<bytes/s>
--dlimit=<bytes/s>

I primarily use it for upload throttling and it seems to work well enough. I did a quick test for download throttling and it seems to work also.


How it works:
It depends on progress stream (since it need to calculate the rate), so we always use progress stream now. To toggle displaying of progress, I changed that to be a parameter in the constructor.

Rate is calculated whenever read() or write() is called. If the rate is greater than the limit, it calculates how long it would need to sleep for to bring it down to the limit, and then it sleeps that amount.


(Un)fortunately the patch is intermingled with my Content-MD5 patch.


Tienshiao
Logged
Scott
Newbie
*
Posts: 2


View Profile
« Reply #3 on: August 09, 2009, 03:48:50 PM »

Try trickle on linux (unix??) http://www.linux.com/archive/articles/61293?tid=100&tid=47&tid=13

It lets you throttle a TCP application, kind of like nice.  I haven't tried it yet, the docs say the limits are in KB not Kb.  So to keep it at 200kbps, I'm gonna try 50KBps to send 15g in 2 days or so.

trickle -u 50 -d 50 s3sync.rb ....

[After some minimal testing, the numbers coming out of ntop do show it works but you'll have to tweak the trickle inputs to get the desired output.   On my system -d 20 -u 20 is reported as roughly 200Kbps in ntop.  50/50 was reported as 500Kbps]
« Last Edit: August 09, 2009, 05:13:04 PM by Scott » Logged
morgant
Newbie
*
Posts: 3



View Profile WWW
« Reply #4 on: November 24, 2009, 12:15:36 PM »

I'd like this feature as well.
Logged
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2013, Simple Machines Valid XHTML 1.0! Valid CSS!