5

I have an application where

Per day 1 millions to 100 millions even more small xml files are getting generated, and i have to load this into one of the S3 bucket as in individual xml files only . That S3 bucket is linked to CloudFront so that our client across the world can access xml files faster .

All is working fine for me except the cost part . The cost of S3 put request is increasing high every day as no of files increase . The files should be pushed as soon as it appears and should be available to access from cloudFront .

Is there any solution that can save my cost here ? The size of the xml files are max 2 kb.

Let me elaborate some more points here that i have tried so far . I thought to combine all small xmls on-prem and then push it to the S3 ,but the issue with this is that we should have some compute in AWS to extract it into again small files because end user will only accept individual xml files . And extracting and creating small files and saving again into S3 will cost even more .

So please feel free to suggest some other system instead of S3 which might be suitable for this use case . I also tried HBASE but running and managing HBASE in AWS is an issue . Also we tried Dynamo DB but that also was costing more .

11
  • Could you please clarify your goal? Are you wanting to combine all files from one day into a single file (one file per day), or are you wanting to combine ALL FILES every uploaded into one file? If you are successfully combining the files together, why are you keeping the original files? Feel free to Edit your question to add more information. Apr 4, 2019 at 4:16
  • @JohnRotenstein i have updated my question please have a look once Apr 4, 2019 at 4:31
  • Sorry, I was confused and thought that you wanted to combine the files fewer, large files. So are you saying that you need to provide all files at all times, and there are over 1 million files added per day? How will your users obtain a listing of the files to know what to download? What are they doing with that number of files? Are multiple users all accessing the same files, or is each file intended for only one user (in which case a Queuing system would be better)? Do they always need access to old (previous day) files? If you can explain how the files will be used, we can answer better. Apr 4, 2019 at 8:40
  • @JohnRotenstein Ok so yes we need to provide all files at all time they may not use it but when required it should be available .The files are even more than 1 millions also in peak time .Regarding listing of files ,we are geneting this files from our data bases and then sending them the pointer i.e details of the file lets say name of that files and then they can access using name from S3 using CloudFront .What are they doing is not in my scope actually but they need in xml file format itself .Also multiple user can donwload same file but not at the same time .Yes older file should be avaible Apr 4, 2019 at 11:36
  • So what is your actual concern? Is your current bill particularly high? It will show the costs broken down into storage, requests, data transfer. Apr 4, 2019 at 12:18

2 Answers 2

1

API calls are charged per object, regardless of its size. Uploading 1-byte costs the same as uploading 1GB. So usually small objects can cause API costs to soar.

In the following scenario you can Batch multiple objects and upload it as a single file to S3 Bucket.

Next you need to write a Lambda Function (Serverless Architecture) to unzip the uploaded file into the same bucket.

Since you have mentioned that all your end users will be accessing the XML files only. So You should upload a Batch file and use Lambda Function to extract all of them into the same bucket.

This will save you PUT Data Transfer cost.

Sample Code for Lambda for unzip

6
  • 2
    So are you saying to zip many small xml files into one and then upload into S3 and once uploaded fire up lambda which will unzip into small files ....But when we unzip and it will create small file again and that also be a put request only even though it is same bucket . Apr 4, 2019 at 4:52
  • There are two aspects, One is Transfer cost and another is a storage cost. Solution which i have mentioned above will definitely curtail down your transfer cost. for Storage cost on those tiny XML's you can use lifecycle to move infrequent accessed files to a different storage class or glacier and then delete. Apr 4, 2019 at 5:15
  • 2
    So actually there is no transfer cost or negligible for punting data into S3 .So transfer cost is not an issue here,Even storage is also not much because we have very small no of xml file hardly 5 gb in day .The real issue is put request into S3 . Apr 4, 2019 at 5:19
  • 2
    Moving small files (less than 128kB) into infrequent access class will actually increase the cost, as the files in IA, when it comes to storage, are billed at a 128kb minimum size. So in standard class for 5 files 10kB each you'll be charged for 50kB of storage, in IA this would be 640. Apr 24, 2021 at 1:33
  • 4
    @YashBindlish How is your solution better than directly writing smaller files? At the end of it, you're still making PUT requests to write unzipped files using Lambda. It should still cost the same, or am I missing something?
    – ashu
    Oct 27, 2021 at 21:08
0

For small files (max 2kb) consider using a database such as DynamoDB.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.