172

I use the following method a lot to append a single row to a dataframe. One thing I really like about it is that it allows you to append a simple dict object. For example:

# Creating an empty dataframe
df = pd.DataFrame(columns=['a', 'b'])

# Appending a row
df = df.append({ 'a': 1, 'b': 2 }, ignore_index=True)

Again, what I like most about this is that the code is very clean and requires very few lines. Now I suppose the recommended alternative is:

# Create the new row as its own dataframe
df_new_row = pd.DataFrame({ 'a': [1], 'b': [2] })
df = pd.concat([df, df_new_row])

So what was one line of code before is now two lines with a throwaway variable and extra cruft where I create the new dataframe. :( Is there a good way to do this that just uses a dict like I have in the past (that is not deprecated)?

2
  • 10
    pandas issue 35407 explains that df.append was deprecated because: "Series.append and DataFrame.append [are] making an analogy to list.append, but it's a poor analogy since the behavior isn't (and can't be) in place. The data for the index and values needs to be copied to create the result." Commented Feb 10, 2022 at 12:15
  • 8
    Came across this warning today. However when I used concat as the alternative I got "cannot concatenate object of type '<class 'list'>'; only Series and DataFrame objs are valid". So frustrating.....
    – Ben Watson
    Commented Feb 13, 2022 at 23:16

9 Answers 9

76

I also like the append method. But you can do it in one line with a list of dicts

df = pd.concat([df, pd.DataFrame.from_records([{ 'a': 1, 'b': 2 }])])

or using loc and tuples for values on DataFrames with incremenal ascending indexes

df.loc[len(df), ['a','b']] = 1, 2

or maybe

df.loc[len(df), df.columns] = 3, 4
4
  • 9
    You can also use ignore_index df = pd.concat([df, pd.DataFrame.from_records([{ 'a': 1, 'b': 2 }])], ignore_index=True) Commented Apr 22, 2022 at 18:50
  • first argument must be an iterable of pandas objects, you passed an object of type "DataFrame". What can I do to solve this? Commented Nov 28, 2022 at 20:07
  • 1
    The arguments are actually inside a list. So the first argument is a list of pandas objects. First item is the original df, second item is the new df generated from records. Commented Nov 29, 2022 at 20:47
  • 1
    I guess this is the best answer out here, but intuitively, this feels wrong because the old dataframe is still overwritten with the new one. Appending in a sense should have greater data security.
    – Rivered
    Commented Dec 4, 2022 at 22:36
72

Create a list with your dictionaries, if they are needed, and then create a new dataframe with df = pd.DataFrame.from_records(your_list). List's "append" method are very efficient and won't be ever deprecated. Dataframes on the other hand, frequently have to be recreated and all data copied over on appends, due to their design - that is why they deprecated the method

7
  • 2
    How do you know that it is deprecated? At pandas.pydata.org/docs/reference/api/… (which currently shows version 1.4.0) I don't see any mention about that. Even at the dev tree I don't see any deprecation warning: pandas.pydata.org/docs/dev/reference/api/…
    – zby
    Commented Feb 2, 2022 at 10:55
  • 7
    I agree ; though when you use append method (with 1.4.0) you run into a "FutureWarning : The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead". You will find the details in the "what's new" page
    – tgrandje
    Commented Feb 4, 2022 at 13:20
  • 1
    @zby the update to the documentation is being dealt with in this pull request: github.com/pandas-dev/pandas/pull/45587 Commented Feb 10, 2022 at 12:21
  • 3
    That is actually the reason they are deprecating df.append. Thank the Pandas maintainers for that. Still, the "new way to do it" should be more proeminent in their docs, for sure.
    – jsbueno
    Commented May 7, 2022 at 16:50
  • 2
    I heard append() will insert dictionaries into their correct corresponding dataframe columns for an existing dataframe with named columns. Given that, will from_records() be able to do the same? Commented Jul 30, 2022 at 9:18
37

If you want to use concat instead:

append

outputxlsx = outputxlsx.append(df, ignore_index=True)

concat

outputxlsx = pd.concat([outputxlsx, df], ignore_index=True)
1
  • 6
    outputxlsx = pd.concat([outputxlsx, df]) is enough since df is a data frame. Commented Jun 14, 2022 at 19:09
10

I was facing a similar issue. The other solutions weren't really working for me. I'm leaving this answer here as an additional possibility to deal with the issue since this is the first google result for certain searches and I myself ended here at least for the second time.

In my case the data is not a dict but just a list of values for a known set of parameters. I want to add the parameter values to a dataframe as rows because this way I can access a series of all the values for one parameter via df[parameter].

I start with an empty DataFrame:

parameters = ['a', 'b', 'c', 'd', 'e', 'f']
df = pd.DataFrame(columns=parameters)

df:

        a   b   c   d   e   f

With append I could add rows very convenient like so:

new_row = pd.Series([1,2,3,4,5,6], index=parameters, name='row1')
df.append(new_row)

df:

        a   b   c   d   e   f
row1    1   2   3   4   5   6

With pd.concat I found this to deliver the same result in very similar way:

new_row = pd.DataFrame([1,2,3,4,5,6], columns=['row1'], index=parameters).T
df = pd.concat((df, new_row))

The key was to create a (1,n) dataframe from the 1d data and then transpose it to match the other dataframe.

1
  • 1
    Or you could create a dictionary out of your list: new = {k: v for k, v in zip(parameters, [1,2,3,4,5,6])} then df = pd.concat([df, pd.DataFrame(new, index=['row1'])]) works Commented Sep 21, 2022 at 6:30
10

For those, like me, who want a descriptive function rather than lots of one-liners, here is an option based on @Rafael Gaitan above.

def append_dict_to_df(df, dict_to_append):
    df = pd.concat([df, pd.DataFrame.from_records([dict_to_append])])
    return df

# Creating an empty dataframe
df = pd.DataFrame(columns=['a', 'b'])

# Appending a row
df = append_dict_to_df(df,{ 'a': 1, 'b': 2 })
3
# Deprecated issue has been resolved

# Creating an empty dataframe
df = pd.DataFrame(columns=['a', 'b'])
print("df columns:", df)

# Appending a row
df = df.append({ 'a': 1, 'b': 2 }, ignore_index=True)
print("df column Values :", df)

# Create the new row as its own dataframe
df_new_row = pd.DataFrame.from_records({ 'a': [3], 'b': [4] })
df = pd.concat([df, df_new_row])
print("pd concat with two df's :", df)
1

You can use the following command

#If your index is a string
df.loc["name of the index"] = pd.Series({"Column 1" : Value1, "Column 2" : Value2,
"Column 3" : Value3, "Column 4" : Value4, ...})

#If your index is a number
df.loc[len(df)] = pd.Series({"Column 1" : Value1, "Column 2" : Value2,
"Column 3" : Value3, "Column 4" : Value4, ...})

Just keep in mind that the changes will be stored in the initial dataframe.

1

I use the following syntax when I need to specify an index value that can have duplicates:

df = pd.concat([df, pd.DataFrame({'a': 1, 'b': 2}, [index_value])])

Similar using lists:

df = pd.concat([df, pd.DataFrame([[1, 2]], [index_value], ['a', 'b'])])

For unique index, I follow @Rafael's answer:

df.loc[index_value, ['a','b']] = 1, 2

or with a dictionary:

df.loc[index_value] = {'a': 1, 'b': 2}
0

If you want to use chained operations in pandas to append new rows, you could use something like this:

import pandas as pd
      
dictionary = {
    "sex": ["male", "female", "male"],
    "age": [10, 25, 36],
    "pclass": ["pclass1", "pclass3", "pclass2"],
    "survived": [True, False, False]
}
df_dict = pd.DataFrame(dictionary)

new_row = pd.DataFrame({"sex": "male", "age": 40, "survived": False, "name": "Alex"}, index=[0])

df= (
  df_dict
  .assign(name=['Alice', 'Bob', 'Charlie'])
  .drop("pclass", axis=1)
  .pipe(lambda df_: pd.concat([df_, new_row], ignore_index = True))
)

For your example, I would use a custom append_row function.

import pandas as pd

def append_row(df1, d):
    df2 = pd.DataFrame(d, index=[0])
    return pd.concat([df1, df2], ignore_index = True)

df = (
  pd.DataFrame(columns=['a', 'b'])
  .pipe(append_row, {'a': 1, 'b': 2 })
)
2
  • 1
    This breaks for several reasons, can you please fix it: you need to remove the stale line .drop("pclass", axis=1), also the assignment of df_dict doesn't work in pandas 2.0. Please retest it.
    – smci
    Commented Sep 9, 2023 at 1:44
  • The first part was only an example, not a complete example. Yoy can see both examples working at this kaggle notebook using pandas 2.0.3 here: kaggle.com/code/jordirosell/pandas-pipe-to-append-new-rows
    – jrosell
    Commented Sep 10, 2023 at 13:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.