For our project, we have a spark application running on an EMR cluster - which runs on schedule using AWS Data Pipelines. (i.e the instances are shut down after the application/job finishes). I am currently looking at alternatives to produce custom metrics (i.e. to publish business metrics)
The best solution I found so far is to use AWS Cloudwatch API's to publish these metrics.
Are there any other alternatives to achieve the same?
Thanks