0

I am using ExcelToCsv nifi processor for conversation of .xlsx files to csv file. Wants to convert bunch of .xlsx files which has data in different format to csv. Once the file get converted to csv ,data is getting changed as below.

FYI.

I have used below property values inside ExcelToCsv processor.

Refered ExcelToCsv nifi processor link

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-poi-nar/1.10.0/org.apache.nifi.processors.poi.ConvertExcelToCSVProcessor/

CSV format:custom

Value separator : comma

Quote character : double quotes

Quote mode : Quote minimal

Here are few points where i observed data got changed.

17.90==>17.900000001

270E+11===> 270000000000

34,45,67,344===>344567344 : for third case,quote character does not get added.

Somebody please let us know why am i getting wrong results in csv ouput file?

How to solve this issue?Or Is there any solution for excel to csv conversion?

2 Answers 2

1
  • Comma (",") is used as separator, so you can't have 34,45,67,344 as single value in your csv file. If you still want to have there comma, you can change file separator from comma to some other character, i.e. pipe ("|"). To change file separator update "Value Separator" filed in ConvertExcelToCSVProcessor nifi processor.
  • Another option is to escape comma, to achieve that you need to play with "Quote Character" and with "Escape Character"
  • To keep values as they were in the excel file, experiment with "Format Cell Values" value.
2
  • Thank you for your reply. i want to keep comma as is in data values and cannot change delimiter to pipe operator. Hence, let me try with second and third option. May 3, 2020 at 10:52
  • I just checked the output by changing the Format Cell=true Quote Character=" Quote Mode : Quote All Values Still , values are not stored as it is in output file. The below are my observations related to that, #DIV/0!===>ERROR:#DIV/0! 1,48,113==>148,113 02-07-2016=>7/2/16 How we can resolve this issue? May 3, 2020 at 12:30
0

Since Nifi does not have processor to support .XLS (older excel) to .CSV conversion, I wrote a python script to perform conversion, and calling it from ExecuteStreamCommand.

While converting excel rows, the Python script also perform cleanup on rows such as add escape character, remove any \n so that resulted CSV won't fail at ValidateRecord or ConvertRecord processor!

Give it a try (need to tweak) and do let us know that whether it's useful in your case!

import csv
import os
import sys
from io import StringIO, BytesIO
import pandas as pd
import xlrd
from pandas import ExcelFile

wb = xlrd.open_workbook(file_contents=sys.stdin.read(),logfile=open(os.devnull, 'w'))
excel_file_df = pd.read_excel(wb, sheet_name='Sheet1', index=False, index_col=0, encoding='utf-8',engine='xlrd')

#flowfile_content = ExcelFile(BytesIO(sys.stdin.read()))
#excel_file_df = pd.read_excel(flowfile_content, sheet_name='Sheet1', index=False, index_col=0, encoding='utf-8')

csv_data_rows = []
header_list = list(excel_file_df.columns.values)
temp_header_list = []

for field in header_list:
    temp = '"' + field +  '"'
    temp_header_list.append(temp)

header_row  = ','.join([str(elem) for elem in temp_header_list])
csv_data_rows.append(header_row)
is_header_row = True
for index, row in excel_file_df.iterrows():

    if is_header_row :
        is_header_row = False
        continue

    temp_data_list = []
    for item in row :
        #item = item.encode('utf-8', 'ignore').decode('utf-8')
        if hasattr(item, 'encode'):
            item = item.encode('ascii', 'ignore').decode('ascii')

        item = str(item)
        item = item.replace('\n', '')
        item = item.replace('",', '" ')
        if item == 'nan':
            item=''
        temp = '"' + str(item) + '"'
        temp_data_list.append(temp)

    data_row = ','.join([str(elem) for elem in temp_data_list])
    data_row = data_row
    csv_data_rows.append(data_row)

for item in csv_data_rows:
    sys.stdout.write("%s\r\n" % item)

ExecuteStreamCommand Processor Configuration

enter image description here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.