61

I've written a bunch of code on the assumption that I was going to use Numpy arrays. Turns out the data I am getting is loaded through Pandas. I remember now that I loaded it in Pandas because I was having some problems loading it in Numpy. I believe the data was just too large.

Therefore I was wondering, is there a difference in computational ability when using Numpy vs Pandas?

If Pandas is more efficient then I would rather rewrite all my code for Pandas but if there is no more efficiency then I'll just use a numpy array...

4
  • 7
    This is probably too broad a question to be useful. pandas provides a bunch of C or Cython optimized routines that can be faster than numpy "equivalents" (e.g. reading text). For something like a dot product, pandas DataFrames are generally going to be slower than a numpy array since pandas is doing a lot more stuff aligning labels, potentially dealing with heterogenous types, and so on. Commented Feb 5, 2014 at 3:25
  • @TomAugspurger Hmmmm okay...is there somewhere I can read about what it excels at vs what it is less optimized for? Commented Feb 5, 2014 at 6:12
  • I'm not sure of a single source for that. I could be glib and say do it yourself :). Profiling can be really important. This doesn't directly answer your question but may be useful anyway. Commented Feb 5, 2014 at 13:32
  • What sort of difference? Capacity difference, performance difference (memory/CPU/parallelism/both?), algorithmic difference, accuracy difference (float vs double, int vs int64), row-major vs column-major...? Please add specifics.
    – smci
    Commented Apr 23, 2019 at 23:47

4 Answers 4

37

There can be a significant performance difference, of an order of magnitude for multiplications and multiple orders of magnitude for indexing a few random values.

I was actually wondering about the same thing and came across this interesting comparison: http://penandpants.com/2014/09/05/performance-of-pandas-series-vs-numpy-arrays/

1
  • the link addresses pandas series vs numpy arrays only, but do their findings also hold for populating 2-dimensional numpy arrays vs pandas dataframes?
    – develarist
    Commented Jan 16, 2021 at 19:52
13

I think it's more about using the two strategically and shifting data around (from numpy to pandas or vice versa) based on the performance you see. As a recent example, I was trying to concatenate 4 small pickle files with 10k rows each data.shape -> (10,000, 4) using numpy.

Code was something like:

n_concat = np.empty((0,4))
for file_path in glob.glob('data/0*', recursive=False):
    n_data = joblib.load(file_path)
    n_concat = np.vstack((co_np, filtered_snp))
joblib.dump(co_np, 'data/save_file.pkl', compress = True)

This crashed my laptop (8 GB, i5) which was surprising since the volume wasn't really that huge. The 4 compressed pickled files were roughly around 5 MB each.

The same thing, worked great on pandas.

for file_path in glob.glob('data/0*', recursive=False):
    n_data = joblib.load(sd)
    try:
        df = pd.concat([df, pd.DataFrame(n_data, columns = [...])])
    except NameError:
        df = pd.concat([pd.DataFrame(n_data,columns = [...])])
joblib.dump(df, 'data/save_file.pkl', compress = True)

One the other hand, when I was implementing gradient descent by iterating over a pandas data frame, it was horribly slow, while using numpy for the job was much quicker.

In general, I've seen that pandas usually works better for moving around/munging moderately large chunks of data and doing common column operations while numpy works best for vectorized and recursive work (maybe more math intense work) over smaller sets of data.

Moving data between the two is hassle free, so I guess, using both strategically is the way to go.

13

In my experiments on large numeric data, Pandas is consistently 20 TIMES SLOWER than Numpy. This is a huge difference, given that only simple arithmetic operations were performed: slicing of a column, mean(), searchsorted() - see below. Initially, I thought Pandas was based on numpy, or at least its implementation was C optimized just like numpy's. These assumptions turn out to be false, though, given the huge performance gap.

In examples below, data is a pandas frame with 8M rows and 3 columns (int32, float32, float32), without NaN values, column #0 (time) is sorted. data_np was created as data.values.astype('float32'). Results on Python 3.8, Ubuntu:

A. Column slices and mean():

# Pandas 
%%timeit 
x = data.x 
for k in range(100): x[100000:100001+k*100].mean() 

15.8 ms ± 101 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

# Numpy
%%timeit 
for k in range(100): data_np[100000:100001+k*100,1].mean() 

874 µs ± 4.34 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Pandas is 18 times slower than Numpy (15.8ms vs 0.874 ms).

B. Search in a sorted column:

# Pandas
%timeit data.time.searchsorted(1492474643)                                                                                                                                                               
20.4 µs ± 920 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

# Numpy
%timeit data_np[0].searchsorted(1492474643)                                                                                                                                                              
1.03 µs ± 3.55 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Pandas is 20 times slower than Numpy (20.4µs vs 1.03µs).

EDIT: I implemented a namedarray class that bridges the gap between Pandas and Numpy in that it is based on Numpy's ndarray class and hence performs better than Pandas (typically ~7x faster) and is fully compatible with Numpy'a API and all its operators; but at the same time it keeps column names similar to Pandas' DataFrame, so that manipulating on individual columns is easier. This is a prototype implementation. Unlike Pandas, namedarray does not allow for different data types for columns. The code can be found here: https://github.com/mwojnars/nifty/blob/master/math.py (search "namedarray").

4
  • What is " large numeric data" Millions? Hundreds of thousands? Thank you :) Commented Aug 26, 2020 at 9:11
  • I performed comparisons on 1-10M rows of data, several columns, like in the examples above. Commented Aug 27, 2020 at 11:57
  • That's a nice comparison, but I think it is incomplete to say the least. What if we have 200 columns (common use case), and we slice that? Obviously, these columns won't be a numpy matrix of shape (200, ...), but 200 variables, grouped together in a Python object. I would very much like to see that comparison
    – Gulzar
    Commented May 23, 2021 at 10:47
  • 1
    Also what about the cost of converting to numpy for the calculation?
    – Gulzar
    Commented May 23, 2021 at 10:58
0

If you are doing array slicing and appending use numpy it's a lot faster then pandas. Like one time i had function that was creating features and target for a multivariate time series problem using sliding window approach. At first i used pandas dataframe to create it then i used numpy array. The difference was too much.

# The pandas function 
def make_data(dataframe: pd.DataFrame, ws: int) -> Tuple[List, List]:
    features = [dataframe.iloc[i:i+ws, :-1].values for i in range(len(dataframe) - ws)]
    targets = [dataframe.iloc[i+ws, -1] for i in range(len(dataframe) - ws)]
    return features, targets

start_time = time.time()
features, targets = make_data(arranged_dataframe, ws=ws)
end_time = time.time()

print(f"Time taken: {end_time - start_time} seconds")

# the numpy function
def make_data(dataframe: pd.DataFrame, ws: int) -> Tuple[np.ndarray, np.ndarray]:
    data = dataframe.to_numpy()
    num_samples = len(data) - ws

    
    features = np.empty((num_samples, ws, data.shape[1]), dtype=data.dtype)
    targets = np.empty(num_samples, dtype=data.dtype)
    
    for i in range(num_samples):
        features[i] = data[i:i+ws, ]
        targets[i] = data[i+ws, -1]
    
    return features, targets


start_time = time.time()
features, targets = make_data(arranged_dataframe, ws=16)
end_time = time.time()
print(f"Time taken: {end_time - start_time} seconds")

output:

Time taken: 0.25033116340637207 seconds
Time taken: 1020.9403311634064 seconds

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.