I'm looking to do a ks_2samp() test on many columns in a pandas dataframe, where each column is split into the two samples i want to compare (idea being to try test for distribution changes in the columns between an initial 'baseline' period and a following 'focus' period).
I've been trying various approaches to see how fast i can get it as i'm typically going to be doing it on 500-1000 columns each with maybe 1000-10000 rows.
I decided to try cython and just loop over the numpy arrays but after a few hours reading docs and trying to adapt some examples, i don't actually see much speed up.
I'm just wondering if i'm missing something obvious or if maybe cython just not going to help me much in this case (my gut is i'm missing something obvious - this is my first time playing with Cython).
Here is a google collab notebook with a reproducible example to show what i mean.
My Cython function looks like this (adapted a lot from the tutorial here):
%%cython
import numpy as np
cimport numpy as np
cimport cython
from scipy.stats import ks_2samp
DTYPE = np.double
cpdef cy_ks_np(double[:, :] arr_a, double[:, :] arr_b):
cdef double k, p
cdef Py_ssize_t i
cdef Py_ssize_t m = arr_a.shape[1]
result = np.zeros((m, 2), dtype=DTYPE)
cdef double[:, :] result_view = result
for i in range(m):
k, p = ks_2samp(arr_a[:,i], arr_b[:,i])
result_view[i,0] = k
result_view[i,1] = p
return result
Really i am just trying to loop over the ks_2samp() lots of times and try give Cython some types (i was, probably naively, hoping this would magically give some speedup from the research i had done).
But when i compare it to a few other approaches i've tried (all in the collab notebook) i don't see much speedup.
Here is what i mean by the timings i am seeing compared to some other approaches i made:
Any pointers on where i might be going wrong here or what a better approach to doing the same scipy function on all columns in a df might be?
I looked a bit at Numba but don't think that can be used on ks_2samp().
cython
isn't compiling or otherwise changingks_2samp
. Looks like the calls to that function dominate the time, not iteration mechanism.