0

Imagine you have a historical data and every day a couple of million rows of data gets added to it. There is a need to process the whole data on a daily basis and update variables. How would you approach this problem using Big data platform?

Happy to provide more details if needed.

5
  • What "Big data platform" did you have in mind? HDFS will happily store your data, and Spark will happily process it Nov 11, 2017 at 0:09
  • I was thinking to have Hortonworks as Big Data Platform. But the challenge is that I need to perform the aggregation process on the whole dataset on a daily basis. Nov 15, 2017 at 14:16
  • Why exactly is that a challenge? Setup a daily process to do whatever you want. Hortonworks provides Oozie for this purpose Nov 15, 2017 at 14:34
  • Size of the data, currently using RDBMS platform it takes almost 2 days to refresh and do aggregation on a weekly basis. The goal is to this on a daily basis using Hadoop platform. Does it make sense? Nov 15, 2017 at 15:01
  • If you want fast aggregations, I might recommend Solr or Elasticsearch instead. But, sure, any distributed processing framework will be quicker than a single-threaded table scan. Nov 15, 2017 at 16:23

1 Answer 1

1

Try very hard not to reprocess the whole 10B rows... I don't know what exactly you are looking for in that large of a dataset, but there is very likely a statistical model in which you can keep summary information, and just reprocess the incremental against that.

cricket_007 is right though, HDFS and Spark are likely your first tools of choice.

1
  • It is more like having a billion rows of raw data and the business requirement is to perform aggregation process on data on a daily basis for modelling purposes. Nov 15, 2017 at 14:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.