I'm using boost::python in a shared library plugin to run some python code. The shared library is loaded as a plugin in run-time by my main program using boost::dll::shared_library API. i.e. my main program does not link with the shared library plugin. My code runs on Ubuntu 20.04.
I'm linking my shared library with Python:
my_shared_lib/CMakeLists.txt:
find_package(Python3 COMPONENTS Development NumPy REQUIRED)
target_link_libraries(${PROJECT_NAME} PRIVATE ${Python3_LIBRARIES} Python3::NumPy)
When running the following code from my shared library I get an ImportError:
#include <boost/python.hpp>
Py_Initialize();
namespace np = boost::python::numpy;
np::initialize(); //ImportError here
I get the following error: ImportError: /home/myuser/.local/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyObject_SelfIter
I verified I have /usr/lib/x86_64-linux-gnu/libpython3.8.so on my LD_LIBRARY_PATH.
When linking my main program with Python, the program OK runs w/o the ImportError main_program/CMakeLists.txt:
find_package(Python3 COMPONENTS Development NumPy REQUIRED)
target_link_libraries(${PROJECT_NAME} PRIVATE ${Python3_LIBRARIES} Python3::NumPy)
My conclusion here is that loading the shared library plugin in run-time does not seem to load libpython3.8.so, while loading libpython3.8.so in the main program makes it available for the shared library plugin.
If I manually load libpython3.8.so in my shared library like this
#include <boost/python.hpp>
#include <dlfcn.h>
dlopen("/usr/lib/x86_64-linux-gnu/libpython3.8.so", RTLD_LAZY | RTLD_GLOBAL);
Py_Initialize();
namespace np = boost::python::numpy;
np::initialize();
I get a different error further along the way:
Traceback (most recent call last):
File "/home/myuser/.local/lib/python3.8/site-packages/numpy/core/init.py", line 23, in
from . import multiarray
File "/home/myuser/.local/lib/python3.8/site-packages/numpy/core/multiarray.py", line 10, in
from . import overrides
File "/home/myuser/.local/lib/python3.8/site-packages/numpy/core/overrides.py", line 6, in
from numpy.core._multiarray_umath import (
ImportError: PyCapsule_Import could not import module "datetime"
How can I make my shared library plugin load libpython3.8.so automatically when it's loaded?
*** EDIT ***
Using gdb I see my initial assumption was wrong. When the main app loads my shared library plugin, it also loads libpython3.8.so.1.0.
Loaded Loaded '/home/myuser/.vs/mainapp/build/plugins/libap_python_detector.so'. Symbols loaded.
Loaded '/usr/local/3rdparty/hpc/5.13.0-39_AMD_EPYC/boost176/build/lib/libboost_numpy38.so.1.76.0'. Symbols loaded.
Loaded '/lib/x86_64-linux-gnu/libpython3.8.so.1.0'. Symbols loaded.
Loaded '/usr/local/3rdparty/hpc/5.13.0-39_AMD_EPYC/boost176/build/lib/libboost_python38.so.1.76.0'. Symbols loaded.
So what's wrong loading numpy?