S3Sync.net
February 02, 2014, 01:25:35 PM *
Welcome, Guest. Please login or register.

Login with username, password and session length
 
  Home Help Search Login Register  
  Show Posts
Pages: [1]
1  General Category / Report Bugs / Can't set group: `chown': bignum too big to convert into `long' (RangeError) on: August 17, 2008, 03:56:36 AM
Hi.

For the first time, I'm trying to sync something from one of my buckets to my local system using s3sync.rb. Up to now, I just uploaded stuff using s3sync.

But now I get this:

Code:
./s3sync.rb:683:in `chown': bignum too big to convert into `long' (RangeError)
from ./s3sync.rb:683:in `send'
from ./s3sync.rb:683:in `updateFrom'
from ./s3sync.rb:393:in `main'
from ./s3sync.rb:735

I'd say that this is related to the closed bug “Owner ID is incorrect”.

Using jets3t, I see the following "Metadata":

  • group: 4294967294
  • owner: 10001
  • permissions: 33204
I synced the file from a Linux system onto S3. On the Linux system, the source was NFS mounted and the file belonged to group "nogroup". I don't use cygwin. So it's not Cygwin related.

I see that problem with s3sync 1.2.6.

The patch from “Owner ID is incorrect” did not work for me, because my GID is not -1 (ie. 0xFFFFFFFF), but -2 (ie. 0xFFFFFFFF - 1).

I'll attach a patch which fixes this problem (for me). It's against s3sync.rb from s3sync 1.2.6.

Cheers,
Alexander
2  General Category / Questions / Parallel sync on: November 02, 2007, 06:54:51 AM
Hello.

I noticed, that it's somewhat slow to upload stuff to s3. In a self developped program in Python, I'm uploading the contents in many parallel threads. This speeds up the whole upload process A LOT.

But uploading data with s3sync is very slow, as it uploads one file at a time only.

Would it be possible to enhance a future version of s3 so, that it does upload in many threads at once? This would make the sync process so much faster.

Thanks a lot,
Alexander
3  General Category / Questions / Re: Filenames with UTF-8 characters are badly encoded on: September 06, 2007, 04:04:39 AM
Don't actually know what's legal.. I pass this to the conversion class Iconv from the ruby library.

Well, "UTF-8" works and that's all I care for Smiley

Also I don't know how to detect the right answer..

You mean regarding setting S3SYNC_NATIVE_CHARSET automatically to the "correct" value?

If you have a look at how the locale is set up (ie. to what values the environment variables LANG and LC_* are set), you could induce what the "correct" value would be. If it's set to POSIX or C, it would be US-ASCII; if it's set to something like de_DE.utf-8 (or de_DE.utf8), it would be UTF-8. Other possible values would be something like en_US.ISO8859-1 which would be for ISO-8859-1. Etc. pp.. Basically, the locale setting would also tell, what character encoding is used.

HOWEVER: It would still be very useful, if there were a way to override the automatically detected value.

Best regards,
Alexander
4  General Category / Questions / Re: Filenames with UTF-8 characters are badly encoded on: September 05, 2007, 11:54:19 AM
You're refering to S3SYNC_NATIVE_CHARSET? Thx for the hint, I missed that variable.

What are legal values? Is utf-8 legal? Does casing matter (ie. Windows-1252 == WINDOWS-1252)?

However, would it be possible to enhance s3sync so, that it automatically sets the charset, according to the locale?

Thx,
Alexander
5  General Category / Questions / Filenames with UTF-8 characters are badly encoded on: September 05, 2007, 06:06:49 AM
Hello.

I transfered some files/dirs to S3 with s3sync. One of this directory is named "Ausflüge". I transfered files with the --public-read option. My system (Gentoo Linux x86) is set to UTF-8.

Now, to be able to access files in this "directory", I need to access "Ausfl%C3%83%C2%BCge", ie. "Ausflüge". So it seems that s3sync (or S3 itself?) did not notice that the source filename were UTF-8 encoded.

Is there a way to tell s3sync (S3?) that the filename is UTF-8 encoded?

Thanks,
Alexander
6  General Category / Questions / Re: Cannot send: 403 Forbidden on: August 21, 2007, 05:46:58 AM
I'm getting the same symptoms, and weirdly, s3cmd.rb works correctly and can list files, but I get 403 forbidden when trying to use s3sync.rb

Exactly. With s3cmd, I can do "listbuckets" and other commands just fine:

Code:
$ AWS_ACCESS_KEY_ID=0C5GNE1××××7Y9SRJ5G2 AWS_SECRET_ACCESS_KEY=ETUqgOouvw××××××2HhLv9UubV0BEq5k5M9lJo9t ./s3cmd.rb listbuckets
fooas
sicherung

Or is it, that the Bucketname "mybucket" and/or foldername "/remotefolder" are no good?

Thanks,
Alexander
7  General Category / Questions / Re: Cannot send: 403 Forbidden on: August 18, 2007, 04:40:55 AM
Okay. And how can that happen? Why is s3sync.rb not getting the credentials? is s3sync not compatible with the version of ruby that I'm using?
8  General Category / Questions / Cannot send: 403 Forbidden on: August 17, 2007, 07:23:14 AM
Hi.

I'm trying to use s3sync v1.1.4 on a Gentoo Linux machine with ruby 1.8.6 (2007-06-07 patchlevel 36) [i686-linux]. I downloaded http://mirbsd.mirsolutions.de/cvs.cgi/src/etc/ssl.certs.shar and unpacked that in ~/Source/s3sync/certs, by cd'ing to that directory and executing the shar file. s3sync is in ~/Source/s3sync/s3sync. Then I ran:

Code:
cd ~/Source/s3sync/s3sync
AWS_ACCESS_KEY_ID=0C5GNE1××××7Y9SRJ5G2 AWS_SECRET_ACCESS_KEY=ETUqgOouvw××××××2HhLv9UubV0BEq5k5M9lJo9t SSL_CERT_DIR=/home/askwar/Source/s3sync/certs ruby s3sync.rb -r --ssl --delete -v /home/askwar/Source/s3sync/ mybucket:/remotefolder
S3 command failed:
list_bucket mybucket max-keys 200 prefix /remotefolder delimiter /
With result 403 Forbidden
S3 ERROR: #<Net::HTTPForbidden:0xb787df18>
s3sync.rb:292:in `+': can't convert nil into Array (TypeError)
        from s3sync.rb:292:in `s3TreeRecurse'
        from s3sync.rb:348:in `main'
        from ./thread_generator.rb:79:in `call'
        from ./thread_generator.rb:79:in `initialize'
        from ./thread_generator.rb:76:in `new'
        from ./thread_generator.rb:76:in `initialize'
        from s3sync.rb:269:in `new'
        from s3sync.rb:269:in `main'
        from s3sync.rb:708

Any ideas about why it failed like this?

Thanks a lot,
Alexander
Pages: [1]
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2013, Simple Machines Valid XHTML 1.0! Valid CSS!