s3sync script for EC2 windows instance

[natron65]

could someone tell me where i can find an s3sync script that i can use on my EC2 windows instance? Here is background on what i'm facing:

have done manual upload to S3 using AWS import/Export
now need to move 500 GB plus to EBS storage
when i run s3sync from command line in windows it eventually errors out with one error code or another (directory, encoding, etc...).

i need a script that will transfer entire bucket from S3 to EC2 EBS, if error is encountered, the script should be smart enough to pass over the file that causing the problem and continue syncing. would be really nice if the script wrote the errors to a txt file so i would know where problem was encountered.

i am not too concerned with skipped files, i just need the majority of files to be transferred over to EBS before i can move on to the next phase of our project.

i am a microsoft network engineer as as such do not have any experience with Ruby and very little experience with linux.

we on a very strict time table and this part of project already 1 week behind schedule. any assistance/advice would be greatly appreciated.

Thank You,
Nate

[ferrix]

I could probably offer advice about errors, if you post some details.

s3sync doesn't have a way to skip errors and keep going.  But you could exclude things which fail and start it again.  Also it only transfers one thing at a time.  It seems like you could use something more brute force.

[natron65]

thanks Ferrix,

based on my research, which may not be comprehensive, s3sync is the best app i know of for transfering between S3 and EC2 because it faster than others when transfers. Correct?

do you know of app that can do what do, as you say, "with brute force", and has similar transfer speeds as s3sync?

here are errors I been getting:

C:/s3sync/s3sync.rb:300:in `iconv': "\342\200\231sNWSeafoodsQT"... (Iconv::IllegalSequence)

AND

C:/s3sync/s3sync.rb:638:in `initialize': No such file or directory - D://server image/sqlsvr/d/NTSYS2/Shared/And/.NET for VFP/decks/Sample Code/DataAccessInDotNet/DataAccessInDotNet/DataAccessInDotNetDXCore.Solution/AssemblyCache/297a3442524fae1dee22f6718b5aae8f.ca
che.s3syncTemp (Errno::ENOENT)




thanks again,

Nate

[ferrix]

Well I'm not sure if it's any faster or better than other tools.. it used to be a long time ago but it has not been worked on very actively for a couple years.  Other tools may very well have surpassed it by now.  Shop around!

For the directory thing, you can try the --make-dirs argument to have the local machine make directories when they don't exist.  That might be the reason for that error. 

For the Iconv error, do you have your S3SYNC_NATIVE_CHARSET set properly (to Windows-1252)?  If so, then maybe the file names are screwy on the S3 side, i.e. if the tool used to upload them did a bad job translating them into UTF-8 to be compliant with S3.  The iconv error has to do with converting strings between different character sets