I am rewriting communication bottleneck. I need to send sub-arrays of 3d numpy
arrays.
But passing the sub-array directly to MPI.Send()
fails with:
ValueError: ndarray is not contiguous
Creating new datatype by MPI.Datatype.Create_vector
does not help - it fails from the same reason.
Simplified example:
a = numpy.zeros([9,9])
sub = a[3:5, 3:5]
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
t = MPI.DOUBLE.Create_vector(2, 2, 9)
t.Commit()
if rank == 0:
sub.flat[:] = range(1,9)
comm.Send([sub, t], dest=1)
else:
comm.Recv([sub, t], source=0)
In actual code I use asynchronous sends/receives. Currently I solve it by copying the sub-array to temporary array with contiguous memory layout.
Problem is that the buffers tend to be pretty large and it eats all the memory and available swap.
I think that creating strided datatype is a way to go, but as I do not have access to the original array with contiguous memory I cannot create the strided buffer.
Also using the lower case version of send/recv is not an option, because as I said I need speed and the data are big.
Currently only idea I have is to create C module extension, where I do all the pointer calculation and return back numpy array with access to the contiguous memory segment containing my sub-array.