1

I am wondering if there is a way to compare the file content format of comma separated value (csv) files in python. I have a script that takes in an input csv file and gives generated output. I tried running the script on a given set of files, but got an error message with the files I was given. To test if the script was the issue, I ran the script using a different set of files with the same format, which worked. This leads me to believe that the one of the files is formatted incorrectly. I tried checking for formatting visually, but there are too many columns and rows to check for formatting issues and column data types.

An example:

File_1.csv content contains:

Field_1,Field_2,Field_3, Field_4, ...
ABC, 2012, CH4, 31231.123, ...
ABC2, 20134, H20, 3234.3432, ...
..., ..., ..., ..., ...

File_2.csv content contains:

Field_1, Field_2, Field_3, Field_4, ...
BBC, 324, OH, 323.232, ...
BBC2, 2112, HCL, 23.2324, ...
..., ..., ..., ..., ...

The idea would be two use the formatting of file_1.csv to make sure file_2.csv has the same format and data types. It would essentially check if file two is following the same format as file one:

<Str>, <Str>, <Str>, <Str>, ...
<Str>, <int>, <Char>, <float>, ...
<Str>, <int>, <Char>, <float>, ...
<Str>, <int>, <Char>, <float>, ...

Assuming that File one has correct column name format and row data types. Is there a program/script has been written that compares file format and data types? Or, would I have to do this from scratch?

2
  • I tried running the script on a given set of files, but got an error message Did the error message give any clues as to what the exact error was? Sep 10, 2018 at 2:32
  • Hi there @JohnGordon, I was able to figure out the how to write it. I did encounter an issue with regex pattern matching if you want to take a look at Mauricio Martinez question/post.
    – SIO
    Sep 17, 2018 at 0:05

1 Answer 1

0

You can do this yourself pretty quick. I'm assuming you're getting an error while trying to type cast one of the fields to an int/char or something. You can use a try-except statement to try and pin-point the error. Here's some pseudo code:

for line in file:
    # separate the fields
    fields  = line.split(',')
    try:
        field1 = int(fields[0])
    except:
        # you had an error

Obviously if you have a lot of fields this could be tedious, but I'm assuming you already have something like this written in your code.

1
  • Hi there @MauricioMartinez, I understand the line.split(',') in the code above. I tried an implementation of it. However, I encountered an issue with re.split(r',(?![^"]")',content1.iloc[0][0])). I tried using a look ahead to avoid commas in double quotes like the following: ,"1,4-Benzenediol; Hydroquinol; p-Quinol", I works for the following ',"2,6-Diamino-4-hexenoic acid, 9CI",', but not the latter. Any help would be much appreciated. I am positive I did not write the look ahead regex correctly, and trying to figure it out.
    – SIO
    Sep 17, 2018 at 0:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.