28

I'm working on a project that incorporates file storage and sharing features and after months of researching the best method to leverage AWS I'm still a little concerned.

Basically my decision is between using EBS storage to house user files or S3. The system will incorporate on-the-fly zip archiving when the user wants to download a handful of files. Also, when users download any files I don't want the URL to the files exposed.

The two best options I've come up with are:

  1. Have an EC2 instance which has a number of EBS volumes mounted to store user files.

    • pros: It seems much faster than S3, and zipping files from the EBS volume is straight forward.
    • cons: I believe Amazon caps how much EBS storage you can use and there is not as redundant as S3.
  2. After files are uploaded and processed, the system pushes those files to an S3 bucket for long term storage. When files are requested I will retrieve the files from S3 and output back to the client.

    • pros: Redundancy, no file storage limits
    • cons: It seems very SLOW, no way to mount an S3 bucket as a volume in filesystem, serving zipped files would mean transferring each file to the EC2 instance, zipping, and then finally sending output (again, slow!)

Are any of my assumptions flawed? Is there a better way of managing massive amounts of file storage?

1
  • 1
    You can mount an S3 bucket as a volume. Check out s3fs. I've used it to upload a huge (5GB) zip file to S3, then mounted my bucket as a volume, then unzipped it. It worked like a charm.
    – Edenbauer
    Aug 11, 2012 at 1:46

5 Answers 5

22

If your service is going to be used by an undetermined number of users, it is important to bear in mind that scaleability will always be a concern, regardless of the option adopted, you will need to scale the service to meet demand, so it would be convenient assume that your service will be running in a Auto Scaling Group with a pool of EC2 instances and not a single instance.

Regarding the protection of the URL to allow only authorized users download the files, there are many ways to do this without requiring your service to act as an intermediate, then you will need to deal with at least two issues:

  1. File name predictability: to avoid URL predictability, you could name the uploaded file as a hash and store the original filenames and ownerships in a database like SimpleDB, optionally you can set a http header such as "Content-Disposition: filename=original_file_name.ext" to advise users browser to name the downloaded file accordingly.

  2. authorization: when the user ask to download a given file your service, issue a temporary authorization using Query String Authentication or Temporary Security Credentials for that specific user giving read access to the file for a period of time then your service redirects to the S3 bucket URL for direct download. This can greatly offload your EC2 pool instances, making then available to process other requests more quickly.

To reduce the space and traffic to your S3 bucket (remember you pay per GB stored and transferred), I would also recommend compressing each individual file using a standard algorithm like gzip before uploading to S3 and set the header " Content-Encoding: gzip " in order to make automatic uncompression work with users browser. If your programming language of choice is Java, I suggest taking a look at the plugin code webcache-s3-maven-plugin that I created to upload static resources from web projects.

Regarding the processing time in compressing a folder, you will frequently be unable to ensure that the folders are going to be compressed in short time, in order to allow the user to download it immediately, since eventually there could be huge folders that could take minutes or even hours to be compressed. For this I suggest you to use the SQS and SNS services in order to allow asynchronous compression processing, it would work as follows:

  1. user requests folder compression
  2. the frontend EC2 instance creates a compression request in an SQS queue
  3. a backend EC2 instance, consumes the compression request of the SQS queue
  4. the backend instance downloads the files from S3 to a EBS drive, since the generated files will be temporary I would suggest to choose to use at least m1.small instances with ephemeral type disks, which are local to the virtual machine in order to reduce I/O latency and the processing time.
  5. after the compressed file is generated, the service uploads the file to the S3 bucket, optionally setting the Object Expiration properties, that will tell S3 bucket to delete the file automatically after a certain period of time (again to reduce your storage costs), and publishes a notification that the file is ready to be downloaded in a SNS topic.
  6. if the user is still online, read the notification from the topic, and notify the user that the zip file is ready to be downloaded, if after a while this notification did not arrive, you can tell the user that compression is taking longer than expected and the service will notify him by e-mail as soon as the file is ready to be downloaded.

In this scenario you could have two Auto Scaling Groups, respectively frontend and backend, that may have different scaleability restrictions.

2
  • It seems like a great solution, but does it work if he want´s to share content on the fly? I think this backend download/upload process could consume time and users that want to be able to download a group of files would give up.
    – Lelis718
    Feb 17, 2014 at 15:45
  • In case one needs to download a handful set of files the EC2 server, instead of redirecting could retrieve each file from S3 and stream the zip file. I don't think this would be a great problem, since processing time and latency between S3 and EC2 instances are generally faster than user user link. Feb 18, 2014 at 22:22
5

If you are insistent on serving the zip files directly from your EC2 instance using S3 will just be more complicated than storing them locally. But S3 is much more durable than any EC2 storage volumes, so I'd recommend using it anyway if the files need to be kept a long time.

You say you don't want to expose the file URLs directly. If that's just because you don't want people to be able to bookmark them and bypass your service authentication in the future, S3 has a great solution:

1 - Store the files you want to serve (zipped up if you want it that way) in a private S3 bucket.

2 - When a user requests a file, authenticate the request and then redirect valid requests to a signed, temporary S3 URL of the file. There are plenty of libraries in a variety of languages that can create those URLs.

3 - The user downloads the file directly from S3, without it having to pass through your EC2 instance. That saves you bandwidth and time, and probably gives the fastest download possible to the user.

This does expose a URL, but that's probably okay. There's no problem if the user saves the URL, because it will not work after the expiration time you set on it. For my service I set that time to 5 minutes. Since it is digitally signed, the user can't change the expiration time in the URL without invalidating the signature.

2
  • How is s3 more "durable"? Ec2 volumes can persist if you want them to. Usually you aren't deleting ec2 instances anyway, unless you are updating them. If you are using docker with aws, that would take care of any database snapshots anyway? Right? Also you can back up ec2 instances, and EBS. Or am I wrong? Mar 10, 2021 at 0:28
  • EBS is a file system, which is much more complicated than a blob store like S3. According to aws.amazon.com/ebs/features, EBS durability is between 99.8% and 99.999%. S3 durability is 99.999999999% per aws.amazon.com/s3/faqs. You can save EBS snapshots to S3, but that's not the same thing as the live EBS volumes being durable. Mar 11, 2021 at 1:45
2

Using S3 is a better option for.this use case. It scales better and will be simplier. Why are you concerned about it being slow? Transfers between EC2 and S3 are pretty snappy.

1

Some considerations:

  1. EBS Volume costs is several times that of S3.
  2. EBS volume size limits are 16 TB, so that should not be an issue. However, volumes of that size are very expensive.
  3. Make sure that your bucket is located in the same region as your EC2 instances.
  4. Use VPC endpoints to communicate with S3. This is much faster.
  5. Make sure that your EC2 instance type has the network bandwidth that you need. CPU and Network speed goes up with instance size.

I would keep everything on S3, download the files as required to zip them into a package. Then upload the zip to S3 and deliver to the user an S3 Signed URL to download from S3.

You could allow the user to download from your EC2 instance, but lots of users have error problems, retry issues, slow bandwidth, etc. If the zip files are small (less then 100 MB) deliver locally, otherwise upload to S3 and let S3 deal with the user download issues.

Another option would be to create a Lambda function that creates the zip file and stores on S3. Now you don't have to worry about network bandwidth or scaling. The Lambda function could either return to you the S3 URL, which you deliver to the browser, or Lambda could email the customer a link. Look into SES for this. Note: The Lambda file system only has 512 MB of space, memory can be allocated up to 1.5 GB. If you are generating zip files larger than this, Lambda won't work (at this time). However, you could create multiple zip files (part1, part2, ...)

2
  • Excellent point on the Lambda function. Do you have a github link or post for a boiler plate lambda? Mar 10, 2021 at 0:29
  • @EvanErickson - I do not have public code for the Lambda part (I was paid to develop that) but a quick Google search shows a number of implementations. Most languages have good libraries for zipping files. Mar 10, 2021 at 1:06
0

The main question is where are you hosting. Since you said you are using a ec2 instance, which means you are leveraging AWS, I would go for EBS and then EFB if you need to scale.

S3 is great, but IMO it is mainly for if you are hosting your site with a different provider, like Namecheap, etc, and just want to use AWS for a database.

I don't think reliability durability matters much, especially when you consider that you can back up snapshots of your Ec2 and EFB.

I would go solely based on price. See which one is cheaper. If there is a significant performance difference (2-5 seconds USER wait time), I would maybe consider spending more for the faster one.

EFB is a method of scaling and might be cheaper than doing EBS. I believe Amazon recommends using EBS until it gets to a certain size and then switching to EFB.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.