I'm new to use pandas in python whereas I have good knowledge in working with python.
I've two data frames from which I've to get matching records and non matching records into new data frames.
Example :
DF1 :
ID Name Number DOB Salary
1 AAA 1234 12-05-1996 100000
2 BBB 1235 16-08-1997 200000
3 CCC 1236 24-04-1998 389999
4 DDD 1237 05-09-2000 450000
DF2 :
ID Name Number DOB Salary
1 AAA 1234 12-05-1996 100000
2 BBB 1235 16-08-1997 200000
3 CCC 1236 24-04-1998 389999
4 DDD 1237 05-09-2000 540000
And, with primary keys being ID & Name here(in reality the number of keys might vary), I need to get
Match_df :
ID Name Number DOB Salary
1 AAA 1234 12-05-1996 100000
2 BBB 1235 16-08-1997 200000
3 CCC 1236 24-04-1998 389999
Mismatch_df :
ID Name Number DOB Salary
4 DDD 1237 05-09-2000 540000
I've tried all possible ways like
pd.merge(df1, df2, left_on=[ID,Name],right_on=[ID,Name], how='inner')
and this produces all the unique keys that are in both the data frames. But this also produces non matching records.
But I'm getting this as my result :
ID Name Number DOB Salary
1 AAA 1234 12-05-1996 100000
2 BBB 1235 16-08-1997 200000
3 CCC 1236 24-04-1998 389999
4 DDD 1237 05-09-2000 540000
where 4th record is also getting included.
Here, only salary col is varying but in real Time, it may be a list of cols to be compared.
From this, I've to get only matching records to the matched_df and non matching records to the mismatch_df.
Kindly help me out in doing this.
Note: My dataset might be a massive one (100 million records in both datasets) so, please get me an effective approach reducing the time of execution.
Thanks in advance.