S3Sync.net
February 02, 2014, 01:31:58 PM
Welcome,
Guest
. Please
login
or
register
.
1 Hour
1 Day
1 Week
1 Month
Forever
Login with username, password and session length
Home
Help
Search
Login
Register
S3Sync.net
>
General Category
>
Questions
>
Parallel sync
Pages: [
1
]
« previous
next »
Print
Author
Topic: Parallel sync (Read 3918 times)
A.Skwar
Newbie
Posts: 8
Parallel sync
«
on:
November 02, 2007, 06:54:51 AM »
Hello.
I noticed, that it's somewhat slow to upload stuff to s3. In a self developped program in Python, I'm uploading the contents in many parallel threads. This speeds up the whole upload process
A LOT
.
But uploading data with s3sync is very slow, as it uploads one file at a time only.
Would it be possible to enhance a future version of s3 so, that it does upload in many threads at once? This would make the sync process so much faster.
Thanks a lot,
Alexander
Logged
ferrix
Sr. Member
Posts: 363
(I am greg13070 on AWS forum)
Re: Parallel sync
«
Reply #1 on:
November 02, 2007, 05:48:44 PM »
I have nothing against that idea. It should be possible to get a fixed number of worker threads going instead of having the comparator wait on one thing at a time.
Logged
grig
Newbie
Posts: 1
Re: Parallel sync
«
Reply #2 on:
December 11, 2008, 11:53:52 PM »
i have waited 10 days for my folder to transfer to s3sync
* s3 seems to have a cap on a single file upload speed (after googling for "s3 slow")
* s3sync tries to sync the same file it's working on in a separate process, so more control of a single directory parallel process doesn't work as-is
* If there was an INCLUDE for s3sync I could specify a few regexps and break the sync up
i am now shifting my dev time (from my actual task at hand) to making this faster by writing a script that uses s3cmd and my own file tree compare
p.s. I'm transferring about 6,000 FLAC files from a single directory on a remote fedora 4 box
peace,
g
Logged
Pages: [
1
]
Print
« previous
next »
Jump to:
Please select a destination:
-----------------------------
General Category
-----------------------------
=> Announcements
=> Questions
=> General Discussion
=> Report Bugs
===> Closed Bugs
=> Feature Requests
Loading...