I need to create a big file, by merging multiple files scattered in several subfolders contained in an Azure Blob Storage, also a transformation needs to be done, each file contains a JSON array of a single element, so the final file, will contain an array of JSON elements.
The final purpose is to process that Big file in a Hadoop & MapReduce job.
The layout of the original files is similar to this:
folder
- month-01
- day-01
- files...
- month-02
- day-02
- files...