Since other answers are old, I would like to add that pd.append
is now deprecated in favour of pd.concat
with pandas 1.4. Thus, what follows are useful information for people running into this issue today.
To concatenate multiple data frames, you should not use append()
with a for-loop, because it can be computationally expensive and slow. You should create a list of data frames and then use concat()
once on the list. For example:
# Create an empty list of data frames
df_list = []
# Loop over the files and append each data frame to the list
for file in file_folder:
df_list.append(pd.read_csv(file))
# Concatenate the list of data frames into one data frame
df_all = pd.concat(df_list)
This way, you avoid creating intermediate copies of the data frames and reduce the memory usage and execution time.
Example |
Using concat() |
Using append() |
Append two data frames row-wise |
df = pd.concat([df1, df2], axis=0) |
df = df1.append(df2) |
Append two data frames column-wise |
df = pd.concat([df1, df2], axis=1) |
df = df1.join(df2) |
Concatenate multiple data frames row-wise with different columns |
df = pd.concat([df1, df2, df3], axis=0, join='outer') |
df = pd.DataFrame() for df_i in [df1, df2, df3]: df = df.append(df_i) |
Concatenate multiple data frames row-wise with common columns |
df = pd.concat([df1, df2, df3], axis=0, join='inner') |
df = pd.DataFrame() for df_i in [df1, df2, df3]: df = df.append(df_i, join='inner') |
Concatenate multiple data frames with a hierarchical index |
df = pd.concat([df1, df2, df3], axis=0, keys=['one', 'two', 'three']) |
df_dict = {} for key, df_i in zip(['one', 'two', 'three'], [df1, df2, df3]): df_dict[key] = df_i df = pd.concat(df_dict.values(), axis=0, keys=df_dict.keys()) |
As it is clear, concat()
uses more concise syntax and is faster. Thus, no reason to use append()
.
Even if you like the deprecated pd.append
, remember that the pd.concat
function is more efficient and faster than the pd.append
function and has the same options and functionality. Like Matus Dubrava mentioned, append
calls concat
under the hood anyway.
append
essentially callsconcat
.df_list
is already a list, you can just pass it toconcat
directly.concat
is to take a list of frames to concatenate, not just 2-item lists that wouldn't provide any benefit over something likedf.append(*list_of_two_dfs)
.