I'm uploading files via AJAX to Amazon S3 (using the browser File API and storing the actual upload script on Amazon S3 as an iframe to get around the Amazon S3 cross-site issues, courtesy to jquery-file-upload for the idea). I have that working and it works great for small files (< 50 MB or so). However, I'm looking to store extremely large files on Amazon S3. I'd like to store things like a configured virtual machine, which could be 10+ GB in size.
From my understanding of the HTML5 file API, large files can be chunked up into small bits on the client and uploaded. It is then the responsibility of the server code to join the files together and move the file to S3. I understand the concept but am not sure of the best implementation.
I'm using Heroku for the app server and I normally upload files directly to Amazon S3, skipping Heroku's servers completely. However, if I chunk the upload into small bits, I would have to have some code that joins the parts before actually putting it in S3. But Heroku has some limitations on how much data can be used with them, and I don't think that joining a 10 GB file would work out effectively on their servers (not 100% sure but doubtful).
So my current thought is that I have to have a web service app setup on an Amazon EC2 server where my app posts the upload parts to. The EC2 app is then responsible for joining the upload parts and putting the final joined file into S3. Once the file is loaded into S3, S3 sends a response to the original app hosted on Heroku, which then creates a resource that points to the stored file in S3.
Is there any realistic way of getting around having a separate EC2 server to join the files? There's no cost for sending files between EC2 and S3, but I don't want to have to maintain 2 apps to accomplish what I want (main app on Heroku and the file-joining app on EC2).