Files being modified during upload

[professionalnerd]

Hi all,

What's the expected behaviour of s3sync if I try to upload a file and it gets updated during the process?

I only ask as one of the things I plan to do is just keep a sync of /var/log on S3, and while testing the other day it kept hanging on /var/log/messages which I presume was due to it being updated while uploading.  It detected the failure and said it was retrying, however then sat there for some time.  As a crude way to check if it was doing I simply looked for it taking any load in top and found it looking pretty inactive.

Do I have any choice other than to copy / tar it out elsewhere then do the upload?

Thanks,

Jon

[ferrix]

I certainly haven't added any special handling for this.  I recommend you use auto rotation, and exclude anything in that dir that doesn't end with a dot+digit.

[professionalnerd]

Hi ferrix, thanks for the response.

I was going to ask if some sort of file locking would make for a reasonable feature request however, to quote a colleague of mine who's more experienced in that sort of thing, it's apparently 'a bit of a minefield'!  A bit of Googling turns up some info which suggests Ruby has various file locking mechanisms which are fairly easy to implement, but I was unable to tell exactly how any of them work.

For some reason I thought there was some sort of lock that enabled you to read from a file while writes continued to be performed on it, sort of like you were viewing a snapshot of it as it was on opening.  Thinking about that though, that seems unlikely and it's most likely it's something that would be implemented on an application level using temp copies, etc.

If all else fails I guess, I'll have to go with making sure that whatever files I backup are definitely not being written to before syncing them however this is something I've always looked for when it's come to backing up.  It's partially so I can make sure I backup the most up to date data, but also so I can keep backups as simple as possible i.e. a straight forward backup of a whole directory tree, without having to note certain exceptions.

Cheers,

Jon

[lowflyinghawk]

linux:
  man 2 flock
  man 2 fcntl

I don't recall on win32.  the short answer is it depends on the OS and on your current privileges.

suppose you could lock a file whenever you wanted: think about what would happen to other processes that were trying to write to it if your network upload speed suddenly shrank to 300 baud. :-(

[ferrix]

This will not be attempted.

[professionalnerd]

ferrix, short and to the point responses like that usually suggest that someone's been irritated by something they consider to be a stupid  question or unreasonable request, such that it doesn't warrant anything more.  If you revisit my post you'll see that I was aware it might not be an appropriate request and so wasn't directly doing that, I was fully aware that this wasn't a straight forward matter and I was simply looking to try and clarify the context of the issue I'm trying to solve.  So, I apologise if I rubbed you up the wrong way and that's how it came out.

• If it's out of the scope of this application and doesn't belong in it (this is after all a syncing application and I'm looking to backup with it), then fine, I understand that it's something I should not expect of it.
• If it's something that's a very complex issue due to the underlying manner in which the OS deals with file locking then that tells me I'm not likely to find a simple solution and I should stop looking and go with archiving the rotated files like suggested, or archiving them first.
• Even if simply it's something you for your own reasons don't want in the app then again, no problem, it's your work and it's your prerogative so I wouldn't argue it.

But please don't take me for one of those irritating users expecting you to make your application fit my needs or hand hold me through solving my problems. I appreciate the work of everyone who gives their time to open source projects and never at any point feel like they're obliged to do what we users request of them. I'm simply trying to answer some questions I've posed to myself and get a greater understanding of my situation so I can find my way to the most appropriate solution for my needs.

lowflyinghawk, thanks for the pointers.  I've checked the man pages but it's my lack of detailed understanding of the terminology and concepts referred to that's been hampering my investigations.  The logic of my thinking is that if something as simple as tar is able to continue archiving a file as it's contents is on opening but while writes continue on it, it suggests to me that the this is a matter that has a solution within the standard locking mechanisms.  But I imagine there's a lot more to it than that.

Never mind though, I think I'll simply go with archiving inboxes and logs by tarring them first.

Cheers,

Jon

[lowflyinghawk]

tar may be able to continue, but what does it actually do, and in what circumstances?  I can't see that there is one set of rules that will result in the desired result, since the desired result would vary according to the user.   think about a few possibilities: 1) another process opens the file for writing and truncates it to zero; 2) another process appends to the file; 3) another process writes a block into the middle of the file;  4) another process deletes the file. 

I wouldn't make the assumption that tar is simple, after all it's been around for more than 20 years and even today there is active development on some versions of it.  and don't forget, tar and ruby have exactly the same set of system calls available to them, so there is no magic going on here.

as far as S3 is concerned, one issue right off the bat is you need to send the content-length header.  what should it be for a file that is continuing to grow?  do you stop after you stat it the first time?  no matter what happens, eventually what you archive will not agree with what is on the drive, is this acceptable?   for a growing file the real rsync sends however many bytes it initially stats; for a shrinking file I don't  know what happens (it will notice the byte count sent is wrong, but I don't know what happens on the server side).

there *are* a few things s3sync should not do, e.g. hang or throw an unhandled exception ;-).

[ferrix]

ferrix, short and to the point responses like that usually suggest that someone's been irritated by something they consider to be a stupid  question or unreasonable request, such that it doesn't warrant anything more. 

Understood, however in this case I'm just really busy, and wanted to say something rather than leave you guessing what I might be planning to do about it!