0

Non-essential background: I am having a multitude of problems with ctypes pointers that I have so far been able to work around by stripping everything down to a c_void_p on the Python side, then digging out what I need in C via recasting and pointer arithmetic. However, I've finally hit a wall in Python that I can't work around and I'm hoping that the sage answers to this MWE allow me to fix all of my other hacks.

The MWE: In both of the cases below, I successfully define, populate, pass, and return a struct from/to Python to/from a simple C foreign function. In the first case (ff1.py), everything is fine. In the second case (ff2.py), the function returns and successfully dereferences in Python, then Python crashes. The difference between the two cases is whether the struct that is passed out from Python is imported as a variable from a module or whether it is returned from a Python function before being passed to C.

ff.h:

struct dummy
{
  int      dim1;
  int      dim2;
  double * array_ptr;
};

void
foreign_func( struct dummy * ptr_to_dummy );

ff.c:

#include "ff.h"

__attribute__((visibility("default")))
void
foreign_func( struct dummy * d_ptr )
{
  const int N = d_ptr->dim1;
  const int M = d_ptr->dim2;

  for( int i=0; i<N; i++ )
    for( int j=0; j<M; j++ )
      d_ptr->array_ptr[ i*M + j ] *= 3.14159;

  return;
}

struct_wo_funcs.py:

import numpy as np
import ctypes

class dummy_struct_type( ctypes.Structure ):
    _fields_ = [ ( 'dim1',      ctypes.c_int ),
                 ( 'dim2',      ctypes.c_int ),
                 ( 'array_ptr', ctypes.POINTER( ctypes.c_double ) ) ]

n = 12
m = 50
a = np.ones( (n,m), dtype=ctypes.c_double, order='C' )

dummy_struct_instance = \
        dummy_struct_type( ctypes.c_int( n ),
                           ctypes.c_int( m ),
                           a.ctypes.data_as( ctypes.POINTER( ctypes.c_double ) ) )

struct_w_funcs.py:

import numpy as np
import ctypes

class dummy_struct_type( ctypes.Structure ):
    _fields_ = [ ( 'dim1',      ctypes.c_int ),
                 ( 'dim2',      ctypes.c_int ),
                 ( 'array_ptr', ctypes.POINTER( ctypes.c_double ) ) ]

def make_instance( n, m ):
    a = np.ones( (n,m), dtype=ctypes.c_double, order='C' )

    return dummy_struct_type( ctypes.c_int( n ),
                              ctypes.c_int( m ),
                              a.ctypes.data_as( ctypes.POINTER( ctypes.c_double ) ) )

ff1.py:

import numpy as np
import ctypes
import os
from struct_wo_funcs import dummy_struct_type, dummy_struct_instance

fflib = ctypes.CDLL( os.path.abspath( 'ff_lib.so' ) )

fflib.foreign_func.argtypes = [ ctypes.POINTER( dummy_struct_type ) ]
fflib.foreign_func.restype  = None

fflib.foreign_func( ctypes.byref( dummy_struct_instance ) )

print( dummy_struct_instance.array_ptr[19] )

ff2.py:

import numpy as np
import ctypes
import os
from struct_w_funcs import dummy_struct_type, make_instance

fflib = ctypes.CDLL( os.path.abspath( 'ff_lib.so' ) )

fflib.foreign_func.argtypes = [ ctypes.POINTER( dummy_struct_type ) ]
fflib.foreign_func.restype  = None

dummy_struct_instance = make_instance( 12, 50 )

fflib.foreign_func( ctypes.byref( dummy_struct_instance ) )

print( dummy_struct_instance.array_ptr[19] )

I have tried this with a variety of compilers on Linux. I have also changed the export and compiled with gcc/MinGW and cl/MSVC on Windows, all to the same effect. (To be fair, running at the MSVC developer prompt does not result in an outright crash; rather, Windows flinches for a while, then recovers back to the prompt without reporting anything.) Here is a representative Linux crash with gcc v7.5.0 and Python 3.6.9:

$ gcc -fPIC -c ff.c && gcc -fPIC -shared -o ff_lib.so ff.o

$ python3 ff1.py
3.14159

$ python3 ff2.py
3.14159
free(): corrupted unsorted chunks
Aborted (core dumped)

I have valgrinded the heck out of this and can only learn that something is wrong in the guts of Python. Ultimately, I want to be able to use functions in Python to generate the structs that I pass out; that is, I wish ff2.py would not crash. If your answer is simply another work-around (like some magic combination of numpy declarations and byrefs, etc.), that is good enough for me. As far as I know, this is a problem with Python and not C, but if you can get it to work by altering the C side of the FFI, I'd love to see how.

2 Answers 2

1

A ctypes pointer to your array data does not keep the array alive. You can make the first one crash with by just del a.

Create a wrapper for the struct that also keeps the numpy array alive by holding a reference to it.

0

@antti-haapala Sure enough! Here is my working version:

struct_w_wrapper.py:

import numpy as np
import ctypes

class dummy_struct_type( ctypes.Structure ):
    _fields_ = [ ( 'dim1',      ctypes.c_int ),
                 ( 'dim2',      ctypes.c_int ),
                 ( 'array_ptr', ctypes.POINTER( ctypes.c_double ) ) ]

##
##  the wrapper
##   the arguments are obviously tailored to
##   dummy_struct_type, not to an arbitrary
##   "struct_to_be_wrapped"
##
def dont_free_me( struct_to_be_wrapped ):
    class wrapper:
        def __init__( self, dim1, dim2, arr ):
            self.dim1   = dim1  ##  these two ints should persist
            self.dim2   = dim2  ##  without wrapping, but whatever
            self.arr    = arr
            self.struct = \
                    struct_to_be_wrapped( ctypes.c_int( dim1 ),
                                          ctypes.c_int( dim2 ),
                                          arr.ctypes.data_as(
                                              ctypes.POINTER(
                                                  ctypes.c_double )))
        def the_struct( self ):
            return self.struct

    return wrapper
                                                

wrapped_dummy_struct_type = dont_free_me( dummy_struct_type )

def make_instance( n, m ):
    a = np.ones( (n,m), dtype=ctypes.c_double, order='C' )

    return wrapped_dummy_struct_type( n, m, a )

ff3.py:

import numpy as np
import ctypes
import os
from struct_w_wrapper import dummy_struct_type, make_instance

fflib = ctypes.CDLL( os.path.abspath( 'ff_lib.so' ) )

fflib.foreign_func.argtypes = [ ctypes.POINTER( dummy_struct_type ) ]
fflib.foreign_func.restype  = None

wrapped_dummy_struct_instance = make_instance( 12, 50 )
dummy_struct_instance = wrapped_dummy_struct_instance.the_struct()

fflib.foreign_func( ctypes.byref( dummy_struct_instance ) )

print( dummy_struct_instance.array_ptr[19] )

Then, from the prompt:

$ python3 ff3.py 
3.14159

UPDATE: I now know that this is an old, known feature (#12836) in Python ctypes for which numpy's ctypes interface has a built-in workaround. There is a data method which mirrors the problematic data_as behavior in this SO question; there is also numpy's own data_as method with what we want here: "The returned pointer will keep a reference to the array."

Since my problem results in a crash, I started a Python bug (#41883), which has some good discussion if you are interested in implementing your own workaround. This other SO answer is also good reading.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.