not sure what that means but I deleted everything from s3 and started over. Told it to copy again with:

./s3sync.rb --progress /home/tkirby/html/avshare_old/media/ chickenrancher:avshare/media/

Let it copy three or four:

Create node 100.wmv
Progress: 2764800b  839107b/s  85%
Create node 1000.wmv
Progress: 9064448b  243780b/s  95%
...

Stopped it and restarted it:

./s3sync.rb --progress /home/tkirby/html/avshare_old/media/ chickenrancher:avshare/media/

Create node 100.wmv
Progress: 3041280b  593944b/s  94%
Create node 1000.wmv
Progress: 8768512b  354275b/s  92%
...

It always starts from the beginning and never seems to report 100% copied in the progress. That might just be because it exits the progress loop before the last update but it's still copying everything every time. Does the bucket have to be created by s3sync.rb program in order to work? (I'm copying into an existing one)

Where does s3sync.rb store the md5?

Cheers,
-Todd

When I run it with debug and verbose it says this:


local item /home/webmstr/html/avshare_old/media/100.wmv
local node object init. Name:100.wmv Path:/home/webmstr/html/avshare_old/media/100.wmv Size:3227980 Tag:17a9e28de46a6a9634a1625299a8cd23
s3TreeRecurse trainorders avshare/media
Trying command list_bucket trainorders max-keys 200 prefix avshare/media delimiter / with 100 retries left
Response code: 200
prefix found: /
S3 item avshare/media_$folder$
s3 node object init. Name:_$folder$ Path:avshare/media_$folder$ Size:0 Tag:d41d8cd98f00b204e9800998ecf8427e
source: 100.wmv
dest: _$folder$
s3 node object init. Name:100.wmv Path:avshare/media/100.wmv Size: Tag:
Create node 100.wmv

Looks like it's trying to compare the local file with 'S3 item avshare/media_$folder$'
Is _$folder$ some variable that's not getting parsed correctly?

Cheers,
-Todd

The error doesn't happen when I don't specify '--progress'. So it's not transferring my file?

Why a separate command to copy single files? It would be easy enough to stat the source path and determine which the user wants to transfer.

Yeah, umm, wet run the first time, dry run the second to make sure it was gonna pick up where it left of (which it didn't)

./s3sync.rb --dryrun --progress /home/tkirby/html/avshare_old/media/ chickenrancher:avshare/media/
Create node 100.wmv
Create node 1000.wmv
Create node 1001.wmv
Create node 1002.wmv
Create node 1003.mp3
Create node 1004.wmv
Create node 1005.wmv
Create node 1006.wmv
Create node 1008.wmv
Create node 1016.wmv
Create node 1017.wmv
Create node 1018.wmv

--dryrun before the paths, same deal. The files on the server are up to 1419 but it still wants to start at 1000 (the beginning)

Anything else I should try?

I'm trying to transfer a couple thousand movie files to S3 and the s3sync script was running happily for a few hours (and several hundred movies) before it ran into some kind of broken pipe error. Anyways, I went to run the script again (using --dryrun like a good little s3syncer) and it seems like it wants to start all over again. Do I need to add some flag to tell it not to retransfer stuff that already exists in the destination location?

CMD:

 ./s3sync.rb --progress /home/tkirby/html/avshare_old/media/ chickenrancher:avshare/media/ --dryrun

--------------------

I saw a snake eating itself tail first, but it disappeared before I could show anyone.

./s3sync.rb --progress /home/tkirby/html/avshare_old/media/1419.wmv chickenrancher:avshare/media/
Update node
Null stream error:
Skipping avshare/media/1419.wmv: undefined method `closed?' for nil:S3sync::ProgressStream