Those are nodes that s3sync uses to denote a folder/directory.  The md5 is well-known so a list operation can quickly know that it's a directory name and not a file.  They need their own node in order to store permission and owner data. 

Other tools after s3sync came along and used different standards.  Such as not storing any dir nodes, or using a node whose name trails a "/" and has empty contents.  These other tools don't really understand what s3sync is doing with its dir nodes, which is why it shows them to you in a manner that you think they are something strange.  Don't worry, they are just normal.

Like some file or dir that used to be there is now gone?

Would help to see your command lines, and see an output with -d

Usually a 400 error means the client side has done something wrong, so that's why we don't just keep retrying it.  (Retries are done to 500 errors because amazon asks us to). 

If you want to change the range of errors we retry on, go into s3try.rb and change

Code:

break unless ((500...600).include? result.http_response.code.to_i) or forceRetry

to be 400...600 (for example)

No but it definitely needs to.  I'm trying to find some time to do a rewrite.

sync is always "changes" only.  If you want to also remove files from s3 that are no longer on D:, you need --delete too.

Yep that's how I do it.

Once I rewrite s3sync completely and take advantage of copy/rename, we can do it in a more rsync-like way that doesn't require uploading every change 7 times.

By nature sync overwrites older stuff with the newest changes.  The nature of "incremental backup" is to preserve the older state AND save the new stuff off to the side. 

The point being, you want to minimize storage but be able to get access to 7 days of data.  You can't have both with a simple sync-like solution.  I'd recommend 7 different save locations, rotating each day which one you write to.  Otherwise you will only have the latest day of backup.

Like rsync, it detects whether each item needs to be changed.  By default it uses the md5 values.  Sometimes that's too expensive, so md5 checking can be turned off and then it will rely on file size and (if I recall correctly) upload time.

The poor project has been neglected for a couple years because of lack of time.  I plan a complete rewrite, avoiding ruby in favor of either lua and C, or java.  But it has not happened yet.

Well I'm not sure if it's any faster or better than other tools.. it used to be a long time ago but it has not been worked on very actively for a couple years.  Other tools may very well have surpassed it by now.  Shop around!

For the directory thing, you can try the --make-dirs argument to have the local machine make directories when they don't exist.  That might be the reason for that error. 

For the Iconv error, do you have your S3SYNC_NATIVE_CHARSET set properly (to Windows-1252)?  If so, then maybe the file names are screwy on the S3 side, i.e. if the tool used to upload them did a bad job translating them into UTF-8 to be compliant with S3.  The iconv error has to do with converting strings between different character sets

I could probably offer advice about errors, if you post some details.

s3sync doesn't have a way to skip errors and keep going.  But you could exclude things which fail and start it again.  Also it only transfers one thing at a time.  It seems like you could use something more brute force.

Not really I just wish for more time myself.  It makes me sad that this and some other projects are stagnant due to a limited amount of my time.

Nope, this feature is lacking in my tools.  As are a bunch of others these days.  Oh for an infinite amount of free time....

I don't know what I could do to help with this.  The http transfer stuff is all down in the ruby library.

Not actively, but I still want to make a new one.  With a more pluggable design, too.