STDOUT maxes out on 16K?

[mrvanes]

This is funny.
Since the backup takes a lot of time, but I want to monitor the backup process at any given time I redirect stdout and stderr to 2 files like this in the backup script.

./s3sync.rb -rv /media/fotos [somebucket]: >backup.log 2>backup.err

I noticed that the process seemed to stop for no apparent reason (backup.log didn't grow) so I restarted it a couple of times and I was thinking about a bugreport or not untill I extended the stderr output with -d!

./s3sync.rb -rvd /media/fotos [somebucket]: >backup.log 2>backup.err

Now this is interresting, because I can monitor progress of s3sync in backup.err (and I see it does!) but backup.log suddenly stops! That's when I saw the filesize of both redirect files:

4364 -rw-r--r--  1 root root  4452913 Nov 22 15:55 backup.err
  16 -rw-r--r--  1 root root    16384 Nov 22 15:41 backup.log

16384 is exactly 16K

I would think this had something to do with my shell or ruby, but stderr continues way beyond 16K so I suspect a hard limit somewhere in s3sync?
Anyway, I thought it would be worth mentioning it.

update: It doesn't max out on 16K, it just stalls a _very_ long time on 16K...

Martin

[ferrix]

Wow that's news to me!  I can say that my software does not contain any special code or limits here, it just prints to stderr and stdout a line at a time as you'd expect.

[mrvanes]

I have been thinking about this and thought this might have to do with buffering. This could be the responsibility/flaw of the shell (bash), but also Ruby. Is there a way to flush buffers in Ruby, like there is in PHP?