0

I am new to python and have a requirement to compare files from two sources.

Requirement:

I should be able to compare two files and find any differences between two files.The files might be .csv, .dat and .xlsx

ie) 1.Records in file1 and not in file 2
2. Records in file2 and not in file1
3. Changed records between two files

In most of the files there would not be any keys which i can use for my comparison. So I would like to use the entire record as key.

You can find my basic version of the script here and would like to get reviewed.

This script is working for my sample files with 3000 rows in each.
Experts, do you see any problem with my approach? Or do you have a better way of doing this?
Any improvements/suggestions is much appreciated. I am using python 3.6(64 bit)

#script to compare two csv files
import pandas as pd

#read file from source1
src1_df = pd.read_csv(r'C:\Users\Samp\compare\Source1.csv', header=None)
#add a indicator field 'Source'as first field in the dataframe
src1_df.insert(0, 'Source', "SRC1")
#remove duplicates
uniq_src1=src1_df.drop_duplicates(keep='first')


#read file from source2
src2_df = pd.read_csv(r'C:\Users\Samp\compare\Source2.csv', header=None)
#add a indicator field 'Source' as first field in the dataframe
src2_df.insert(0, 'Source', "SRC2")
#remove duplicates
 uniq_src2=src2_df.drop_duplicates(keep='first')


#append the two dataframes horizontally 
full_set = pd.concat([uniq_src1,uniq_src2],ignore_index=True)
#drop duplicates based on the entire row but for the first field 'Source'    
diff_df=full_set.drop_duplicates
                   (full_set.columns.difference(['Source']),keep=False)

#write the output to a csv   
diff_df.to_csv(r''C:\Users\Samp\compare\compare_results.csv',
                                            index=False,encoding='utf-8')
#end of script

File1:

Refrigerators,Barry French,293,457.81
Heavy Gauge Vinyl,Barry French,293,46.71
Holmes HEPA Air Purifier,Carlos Soltero,714,30.94
Bulbs,Carlos Soltero,515,4.43
Avery 52,Carlos Soltero,1412,26.92

File2:

Refrigerators,Barry French,293,457.81
Heavy Gauge Vinyl,Barry French,293,46.71
Holmes HEPA Air Purifier,Carlos Soltero,847,30.94
Floodlight Bulbs,Carlos Soltero,515,4.43
Accessory37,Alan Barnes,2532,-78.96

So The expected output is:

SRC1:Bulbs,Carlos Soltero,515,4.43
SRC2:Floodlight Bulbs,Carlos Soltero,515,4.43
SRC1:Holmes HEPA Air Purifier,Carlos Soltero,714,30.94
SRC2:Holmes HEPA Air Purifier,Carlos Soltero,847,30.94
SRC1:Avery 52,Carlos Soltero,1412,26.92
SRC2:Accessory37,Alan Barnes,2532,-78.96

Thank you so much for your time!

2
  • Why isn't Binder,Barry French,293,46.71 in output?
    – Zero
    Jun 24, 2017 at 20:46
  • Thanks John. Nice Catch. I have corrected my sample data.
    – Samper
    Jun 26, 2017 at 3:25

1 Answer 1

0

You could iterate through dataframes and do row checks?

In [48]: values = []
    ...: for x in range(len(src_1df.index)):
    ...:     if (src_1df.ix[x].tolist() == src_2df.ix[x].tolist()):
    ...:         continue
    ...:     values.append('SRC1:' + str(src_1df.ix[x].values))
    ...:     values.append('SRC2:' + str(src_2df.ix[x].values))
    ...: values
    ...:
Out[48]:
["SRC1:['Holmes HEPA Air Purifier' 'Carlos Soltero' 714 30.940000000000001]",
 "SRC2:['Holmes HEPA Air Purifier' 'Carlos Soltero' 847 30.940000000000001]",
 "SRC1:['Bulbs' 'Carlos Soltero' 515 4.4299999999999997]",
 "SRC2:['Floodlight Bulbs' 'Carlos Soltero' 515 4.4299999999999997]",
 "SRC1:['Avery 52' 'Carlos Soltero' 1412 26.920000000000002]",
 "SRC2:['Accessory37' 'Alan Barnes' 2532 -78.959999999999994]"]
1
  • Thanks!. I would try this. Were you able to check the script which i have in my main post? But do see any problem in my approach?
    – Samper
    Jun 26, 2017 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.