I have the following problem in Azure Data Factory:
In a ADLS I have a CSV file with a linebreak in a value:
A, B, C
a, b, c
a, "b
b", c
This CSV is loaded in a (CSV) Dataset (in ADF) with the following settings; first row is header, quote character double quote ("), column delimiter Comma (,), row delimiter (\r,\n, or \r\n) and escape char backslash ().
The "Preview Data" of the dataset seems to work correct and outputs a table with 2 rows. This is also the output I expect, as the overal structure of the data is preserved.
However, when I try to use this dataset in Mapping Data Flow and select "Data Preview" (in the source node directly), I get the following output:
The linebreak isn't ignored, even as the whole value is between double quotes. The overal structure of the data is now broken as one row is splitted into two.
I get this kind of data when I save an Excel file with linebreaks in some cells as CSV. How should I work around this problem? Should I save the Excel differently, should I try to delete all linebreaks before saving as CSV, or is there a way to let Data Factory solve this problem? Als why does the Preview Data function in a Dataset seems to work correct, where the Data Preview function in Mapping Data Flow does not?