I have an application where
Per day 1 millions to 100 millions even more small xml files are getting generated, and i have to load this into one of the S3 bucket as in individual xml files only . That S3 bucket is linked to CloudFront so that our client across the world can access xml files faster .
All is working fine for me except the cost part . The cost of S3 put request is increasing high every day as no of files increase . The files should be pushed as soon as it appears and should be available to access from cloudFront .
Is there any solution that can save my cost here ? The size of the xml files are max 2 kb.
Let me elaborate some more points here that i have tried so far . I thought to combine all small xmls on-prem and then push it to the S3 ,but the issue with this is that we should have some compute in AWS to extract it into again small files because end user will only accept individual xml files . And extracting and creating small files and saving again into S3 will cost even more .
So please feel free to suggest some other system instead of S3 which might be suitable for this use case . I also tried HBASE but running and managing HBASE in AWS is an issue . Also we tried Dynamo DB but that also was costing more .