2

Suppose I have a couple of terabytes worth of data files that have accumulated on an EC2 instance's block storage.

What would be the most efficient way of downloading them to a local machine? scp? ftp? nfs? http? rsync? Going through an intermediate s3 bucket? Torrent via multiple machines? Any special tools or scripts out there for this particular problem?

3
  • 1
    For sufficiently large data sets, the fastest and cheapest way is to physically ship storage media. See AWS Snowball.
    – JimD.
    Apr 28, 2017 at 6:53
  • True, although I had a feeling that this service is more useful when the data goes into petabytes, where the time to ship a physical drive becomes actually comparable with the time to transfer over internet. A terabyte or so is probably still not that much. None the less, the choice of a protocol, it seems, may turn a several hour transfer into a several day one and vice-versa, hence this question.
    – KT.
    Apr 28, 2017 at 9:22
  • 1
    It is still relevant for 10TB. Look at your bandwidth costs for transferring 10TB out, and then how long will it take to move the data at, say, 100 Mb/s. A 10TB drive sent by FedEx has amazing bandwidth and the FedEx bill is small for the bandwidth.
    – JimD.
    Apr 28, 2017 at 10:31

3 Answers 3

5

As I did not really receive a convincing answer, I decided to make a small measurement myself. Here are the results I got:

enter image description here

More details here.

1
  • 1
    It is hard to add something to your own answer. Thanks for findings. I would only mention that in case you have many small files, you might want to investigate downloading in parallel to fully utilize available bandwidth. But if they are big files, it won't help as much due to less overhead.
    – user8369145
    Jun 13, 2018 at 20:25
4

Please follow these rules:

  • Move as one file, tar everything into a single archive file.
  • Create S3 bucket in the same region as your EC2/EBS.
  • Use AWS CLI S3 command to upload file to S3 bucket.
  • Use AWS CLI to pull the file to your local or wherever another storage is.

This will be the easiest and most efficient way for you.

2
+100

Some more info about this usecase is needed. I hope below concepts are helpfull:

  • HTTP - fast, easy to implement, versatile and has small overhead.
  • Resilio (formerly BitTorrent Sync) - fast, easy to deploy, decentralized, and secure. Can handle transfer interruptions. Works if both endpoints are behind NAT.
  • rsync - old school and well known solution. Can resume transfer and fast in syncing big amounts of data.
  • Upload to S3 and get from there - Upload to S3 is fast. Next You can use HTTP(S) or BitTorrent to get data localy.
10
  • I'm not exactly sure what additional info I could provide. The question is simply about which method would be the fastest one in general, assuming I am ready to spend some hours for setting up the system. I do not think that this answer helps me, though. For example, if I set up HTTP, I presume I'd need to download in parallel? How would it compare to downloading from S3 in parallel or rsync? As for BitTorrent, would I need to set up additional machines to replicate the data before downloading for it to make any sense?
    – KT.
    Apr 26, 2017 at 9:57
  • In general, it seems hard to believe that there are really no good standard answer to a question "how to download files from a remote network" in the year 2017.
    – KT.
    Apr 26, 2017 at 9:57
  • Well, "most efficient" is not a clearly defined criteria. All of mentioned solutions are efficient, while being different. For Resilio, no additional stuff, just a client on both sides.
    – Janusz
    Apr 26, 2017 at 15:06
  • By efficient I assume the total time needed to complete the transfer (assuming no bandwidth limitations on the receiving size and a reasonable budget).
    – KT.
    Apr 27, 2017 at 18:46
  • Would Resilio know how to max out my incoming bandwidth? Is it better than rsync in this respect? Would an intermediate S3 storage step speed things up?
    – KT.
    Apr 27, 2017 at 18:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.