This is an extension to my previous question, Drop duplicates in a subset of columns per row, rowwise, only keeping the first copy, rowwise I also have a similar question here which has a different requirement Drop duplicates in a subset of columns per row, rowwise, only keeping the first copy, rowwise only if every column has the same duplicate
I have the following dataframe. (actual one is around 7 million rows)
import pandas as pd
data = {'date': ['2023-02-22', '2023-02-21', '2023-02-23'],
'x1': ['descx1a', 'descx1b', 'descx1c'],
'x2': ['ALSFNHF950', 'KLUGUIF615', np.nan],
'x3': [np.nan, np.nan, 24319.4],
'x4': [np.nan, np.nan, 24334.15],
'x5': [np.nan, np.nan, 24040.11],
'x6': [np.nan, 75.51, 24220.34],
'x7': [np.nan, np.nan, np.nan],
'v': [np.nan, np.nan, np.nan],
'y': [404.29, np.nan, np.nan],
'ay': [np.nan, np.nan, np.nan],
'by': [np.nan, np.nan, np.nan],
'cy': [np.nan, np.nan, np.nan],
'gy': [np.nan, np.nan, np.nan],
'uap': [404.29, 75.33, np.nan],
'ubp': [404.29, 75.33, np.nan],
'sf': [np.nan, 2.0, np.nan]}
df = pd.DataFrame(data)
If there are more than 3 or more duplicates of a number in any of the columns x3,x4,x5,x6,x7,v,y,ay,by,cy,gy,uap,ubp, I want to to delete the duplicates and only keep one copy, the first column in which the duplicate appears or the column that I can select if that's possible.
The output should look like this,
data = {'date': ['2023-02-22', '2023-02-21', '2023-02-23'],
'x1': ['descx1a', 'descx1b', 'descx1c'],
'x2': ['ALSFNHF950', 'KLUGUIF615', np.nan],
'x3': [np.nan, np.nan, 24319.4],
'x4': [np.nan, np.nan, 24334.15],
'x5': [np.nan, np.nan, 24040.11],
'x6': [np.nan, 75.51, 24220.34],
'x7': [np.nan, np.nan, np.nan],
'v': [np.nan, np.nan, np.nan],
'y': [404.29, np.nan, np.nan],
'ay': [np.nan, np.nan, np.nan],
'by': [np.nan, np.nan, np.nan],
'cy': [np.nan, np.nan, np.nan],
'gy': [np.nan, np.nan, np.nan],
'uap': [np.nan, 75.33, np.nan],
'ubp': [np.nan, 75.33, np.nan],
'sf': [np.nan, 2.0, np.nan]}
The second row shouldn't be affected because there's only 2 copies of the number.
The previous question had the answer,
check = ['x3', 'x4', 'x5', 'x6', 'x7', 'v', 'y', 'ay', 'by', 'cy', 'gy', 'uap', 'ubp']
df.loc[:, check] = df.loc[:, check].mask(df.loc[:, check].apply(pd.Series.duplicated, axis=1))
print(df)
But if I do that, then one of the 75.33 would be deleted. That's not what I want.
I was thinking maybe I can do a for loop per row and then replace the value but I have over 7 million rows of data. Any ideas?