I wanted to achieve an incremental load/processing and store them in different places using Azure Data Factory after processing them, e.g:
External data source (data is structured) -> ADLS (Raw) -> ADLS (Processed) -> SQL DB
Hence, I will need to extract a sample of the raw data from the source, based on the current date, store them in an ADLS container, then process the same sample data, store them in another ADLS container, and finally append the processed result in a SQL DB.
ADLS raw:
2022-03-01.txt
2022-03-02.txt
ADLS processed:
2022-03-01-processed.txt
2022-03-02-processed.txt
SQL DB:
All the txt files in the ADLS processed container will be appended and stored inside SQL DB.
Hence would like to check what will be the best way to achieve this in a single pipeline that has to be run in batches?