S3Sync.net

General Category => Closed Bugs => Topic started by: bbrown on April 28, 2007, 12:09:33 AM



Title: Crash when file is deleted during sync
Post by: bbrown on April 28, 2007, 12:09:33 AM
Hi,

I'm a new user to s3sync. As I write this, I'm doing my first full backup. It has been going for a couple of hours already.

During the sync, I've been cleaning up a few files here and there. This exposed an apparent problem with s3sync: if a file is deleted while s3sync is working, it crashes. Here's the error:

Create node web/users/docs/CC-Document-13
/usr/lib/ruby/1.8/delegate.rb:158:in `method_missing': undefined method `length' for nil:S3sync::ProgressStream (NoMethodError)
        from /usr/lib/ruby/1.8/net/http.rb:1507:in `send_request_with_body'
        from /usr/lib/ruby/1.8/net/http.rb:1496:in `exec'
        from /usr/lib/ruby/1.8/net/http.rb:1044:in `_HTTPStremaing_request'
        from ./HTTPStreaming.rb:45:in `request'
        from ./S3_s3sync_mod.rb:50:in `make_request'
        from ./S3.rb:152:in `put'
        from ./s3try.rb:57:in `S3try'
        from ./s3sync.rb:499:in `updateFrom'
        from ./s3sync.rb:370:in `main'
        from ./s3sync.rb:690


The file web/users/docs/CC-Document-13 was one of the ones I had deleted before s3sync got to it.

Can s3sync be modified so it simply ignores (or prints a warning) if a file has been deleted?


Title: Re: Crash when file is deleted during sync
Post by: lowflyinghawk on April 29, 2007, 04:12:25 PM
I don't know the internals, but presumably it is making up a list and then processing it?  if it processed a file after opening it, it wouldn't matter (on unix-like systems) if you deleted the file, it would back it up anyway because the file doesn't go away until the ref count is zero.  of course it shouldn't barf if opening a file fails ;-).


Title: Re: Crash when file is deleted during sync
Post by: bbrown on April 29, 2007, 05:52:14 PM
No doubt s3sync is compiling a list of files to copy before it actually begins copying. That would account for the long pause before any files start being transferred.

However, I doubt that it is keeping each and every file open. Instead, it is trying to open a file that is non-existent.


Title: Re: Crash when file is deleted during sync
Post by: ferrix on April 29, 2007, 06:05:52 PM
"compile a list" is a bit misleading.  It only keeps a partial list, the minimum of things needed in order to make sure the ordering will match up correctly.  Creating a whole list on drives with a lot of small files sometimes uses up all memory on the machine, so I don't do that.


Title: Re: Crash when file is deleted during sync
Post by: ferrix on June 03, 2007, 12:04:55 AM
By the way, this error should have been caught in 1.1.3
the NoMethodError thing happens when using --progress and the file is not readable on the local side.

This happens when it's removed, or when it just errors out such as when it's a unix pipe or what not.