14

The purpose of my code is to import 2 Excel files, compare them, and print out the differences to a new Excel file.

However, after concatenating all the data, and using the drop_duplicates function, the code is accepted by the console. But, when printed to the new excel file, duplicates still remain within the day.

Am I missing something? Is something nullifying the drop_duplicates function?

My code is as follows:

import datetime
import xlrd
import pandas as pd
#identify excel file paths
filepath = r"excel filepath"
filepath2 = r"excel filepath2"
#read relevant columns from the excel files
df1 = pd.read_excel(filepath, sheetname="Sheet1", parse_cols= "B, D, G, O")
df2 = pd.read_excel(filepath2, sheetname="Sheet1", parse_cols= "B, D, F, J")
#merge the columns from both excel files into one column each respectively
df4 = df1["Exchange Code"] + df1["Product Type"] + df1["Product Description"] + df1["Quantity"].apply(str)
df5 = df2["Exchange"] + df2["Product Type"] + df2["Product Description"] + df2["Quantity"].apply(str)
#concatenate both columns from each excel file, to make one big column containing all the data
df = pd.concat([df4, df5])
#remove all whitespace from each row of the column of data
df=df.str.strip()
df=["".join(x.split()) for x in df] 
#convert the data to a dataframe from a series
df = pd.DataFrame({'Value': df}) 
#remove any duplicates
df.drop_duplicates(subset=None, keep="first", inplace=False)
#print to the console just as a visual aid
print(df)
#print the erroneous entries to an excel file
df.to_excel("Comparison19.xls") 
3
  • hint: read the params df.drop_duplicates(subset=None, keep="first", inplace=False)
    – EdChum
    Sep 29, 2017 at 13:21
  • From a cursory look, you don't save the modifications you do to df when you use the drop_duplicates method. You'll have to set inplace to True or reassign to the same variable name.
    – Alex
    Sep 29, 2017 at 13:22
  • 1
    Possible duplicate of DataFrame.drop_duplicates and DataFrame.drop not removing rows
    – IanS
    Sep 29, 2017 at 13:24

7 Answers 7

30

You've got inplace=False so you're not modifying df. You want either

 df.drop_duplicates(subset=None, keep="first", inplace=True)

or

 df = df.drop_duplicates(subset=None, keep="first", inplace=False)
1
  • Replacing inplace=False with inplace=True worked! Thanks Keith. Sep 29, 2017 at 14:09
15

I have just had this issue, and this was not the solution.

It may be in the docs - I admittedly havent looked - and crucially this is only when dealing with date-based unique rows: the 'date' column must be formatted as such.

If the date data is a pandas object dtype, the drop_duplicates will not work - do a pd.to_datetime first.

1
  • I tried this and it did not work: df['date_time']=pd.to_datetime(df.index). I am still getting non-duplicate rows dropped. Nov 29, 2020 at 9:14
12

If you are using a DatetimeIndex in your DataFrame this will not work

df.drop_duplicates(subset=None, keep="first", inplace=True)

Instead one can use:

df = df[~df.index.duplicated()]

Make sure first the index is not of dtype object but datetime64, which you can check using df.index. You may need to convert the index first using

df = pd.to_datetime(df.index)
4
  • This is not working for me. I have tried making df=df.set_index('date_time') and then df=df[~df.index.duplicated()]. But I am still getting rows dropped that are not duplicated. Nov 29, 2020 at 9:20
  • This also works for multiindex DataFrames with datetime parts
    – chazzmoney
    Oct 20, 2021 at 3:35
  • 2
    @WesleyKitlasten As BAC83 has commented, make sure the index is not of type object but datetime64. You may need to convert the index first with pd.to_datetime(df.index) Jun 13, 2022 at 13:03
  • WORKED FOR ME - 2 steps: 1 - make sure your index is pd.to_datetime(df.index) 2 - Use the df = df[~df.index.duplicated()] Mar 11 at 13:22
11

Might help anyone in the future.

I had a column with dates, where I tried to remove duplicates without success. If it's not important to keep the column as a date for further operations, I converted the column from type object to string.

df = df.astype('str')

Then I performed @Keith answers

df = df.drop_duplicates(subset=None, keep="first", inplace=False)
3
  • 1
    Huge +1, was struggling to make sense of why I was still getting duplicate records. In my case, had a datetime field which I then changed to df['x'] = df['x'].dt.date. This worked, so I didn't have to convert to a string, but wouldn't have tested if this hadn't been pointed out. Thank you! Jan 5, 2021 at 22:08
  • 1
    Glad you got it sorted out :)!. I agree with struggling with date variables so I'm happy to share ideas as I think dates is one of the more tricky aspects of coding.
    – Wizhi
    Jan 7, 2021 at 20:15
  • This saved me. I used df.applymap(str) but of course same result
    – Martin H
    Jan 4, 2022 at 17:19
5

The use of inplace=False tells pandas to return a new dataframe with duplicates dropped, so you need to assign that back to df:

df = df.drop_duplicates(subset=None, keep="first", inplace=False)

or inplace=True to tell pandas to drop duplicates in the current dataframe

df.drop_duplicates(subset=None, keep="first", inplace=True)
1

Not sure if this is a good place to put it. But I recently learned that .drop_duplicates() has to have a match in ALL subsets for dropping a row.

So for deleting multiple based on only the one value i used this code:

no_duplicates_df = df.drop_duplicates(subset=['email'], keep="first", inplace=False)                     # Delete duplicates in email
no_duplicates_df = no_duplicates_df.drop_duplicates(subset=['phonenumber'], keep="first", inplace=False) # Delete duplicates in phonenumber
0

I had the same problem, but a different reason.

After appending one dataframe to another I wanted to de-duplicate based on an id (integer). However, appending changed the type of that column to float and it did not work (see https://github.com/pydata/pandas/issues/6485). I fixed it by running the following before running drop_duplicates:

df = df.astype({'id': 'int64'})

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.