1

This is an extension to my previous question, Drop duplicates in a subset of columns per row, rowwise, only keeping the first copy, rowwise I also have a similar question here which has a different requirement Drop duplicates in a subset of columns per row, rowwise, only keeping the first copy, rowwise only if every column has the same duplicate

I have the following dataframe. (actual one is around 7 million rows)

import pandas as pd

data = {'date': ['2023-02-22', '2023-02-21', '2023-02-23'],
        'x1': ['descx1a', 'descx1b', 'descx1c'],
        'x2': ['ALSFNHF950', 'KLUGUIF615', np.nan],
        'x3': [np.nan, np.nan, 24319.4],
        'x4': [np.nan, np.nan, 24334.15],
        'x5': [np.nan, np.nan, 24040.11],
        'x6': [np.nan, 75.51, 24220.34],
        'x7': [np.nan, np.nan, np.nan],
        'v': [np.nan, np.nan, np.nan],
        'y': [404.29, np.nan, np.nan],
        'ay': [np.nan, np.nan, np.nan],
        'by': [np.nan, np.nan, np.nan],
        'cy': [np.nan, np.nan, np.nan],
        'gy': [np.nan, np.nan, np.nan],
        'uap': [404.29, 75.33, np.nan],
        'ubp': [404.29, 75.33, np.nan],
        'sf': [np.nan, 2.0, np.nan]}

df = pd.DataFrame(data)

If there are more than 3 or more duplicates of a number in any of the columns x3,x4,x5,x6,x7,v,y,ay,by,cy,gy,uap,ubp, I want to to delete the duplicates and only keep one copy, the first column in which the duplicate appears or the column that I can select if that's possible.

The output should look like this,


data = {'date': ['2023-02-22', '2023-02-21', '2023-02-23'],
        'x1': ['descx1a', 'descx1b', 'descx1c'],
        'x2': ['ALSFNHF950', 'KLUGUIF615', np.nan],
        'x3': [np.nan, np.nan, 24319.4],
        'x4': [np.nan, np.nan, 24334.15],
        'x5': [np.nan, np.nan, 24040.11],
        'x6': [np.nan, 75.51, 24220.34],
        'x7': [np.nan, np.nan, np.nan],
        'v': [np.nan, np.nan, np.nan],
        'y': [404.29, np.nan, np.nan],
        'ay': [np.nan, np.nan, np.nan],
        'by': [np.nan, np.nan, np.nan],
        'cy': [np.nan, np.nan, np.nan],
        'gy': [np.nan, np.nan, np.nan],
        'uap': [np.nan, 75.33, np.nan],
        'ubp': [np.nan, 75.33, np.nan],
        'sf': [np.nan, 2.0, np.nan]}

The second row shouldn't be affected because there's only 2 copies of the number.

The previous question had the answer,

check = ['x3', 'x4', 'x5', 'x6', 'x7', 'v', 'y', 'ay', 'by', 'cy', 'gy', 'uap', 'ubp']
df.loc[:, check] = df.loc[:, check].mask(df.loc[:, check].apply(pd.Series.duplicated, axis=1))
print(df)

But if I do that, then one of the 75.33 would be deleted. That's not what I want.

I was thinking maybe I can do a for loop per row and then replace the value but I have over 7 million rows of data. Any ideas?

1
  • Note that duplicate on float values are not ideal, unless you know for sure that your values are limited. Feb 24 at 20:00

1 Answer 1

1

You can stack the data, work with duplicates there and unstack (pivot) back:

s = df.iloc[:,3:].stack().reset_index(name='value')
groups = s.groupby(['level_0','value'])
counts, cumcounts = groups['value'].transform('size'), groups.cumcount()

# verify your condition here - logic might not work as expected
s.loc[counts.ge(3) & (s['level_1'].eq('x6') | (s['level_1'].ne('x6') & cumcounts.gt(0))), 'value'] = np.nan
out = s.pivot(*s).reindex(columns=df.columns, index=df.index)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.