Встраивание Python, которое использует Numpy в C ++, не работает в библиотеке, динамически загружаемой во время выполнения - PullRequest
0 голосов
/ 17 марта 2020

Я хочу встроить Python в C ++ в Ubuntu Linux через динамически загружаемую библиотеку (dlopen / dlsym). Когда я встраиваю модуль Python, который импортирует numpy через один статически связанный исполняемый файл C ++, все работает нормально. Когда я использую динамически загружаемую библиотеку, не пытаясь импортировать numpy, все работает нормально. Однако, когда я пытаюсь импортировать numpy из библиотеки, которая была загружена динамически, происходит сбой.


Для начала, вот простой пример без динамических c библиотек, которые работают

test_stati c. cpp

#include <iostream>
#include <Python.h>

int main(){

  Py_Initialize();

  PyObject* pName = PyUnicode_FromString("mymodule");
  PyObject* pModule = PyImport_Import(pName);
  if (!pModule){
    std::cout << "PyImport_Import() failed for " << "test" << std::endl;
    PyErr_Print();
    exit(-1);
  }

  PyObject* pDict = PyModule_GetDict(pModule);
  if (!pDict){
    std::cout << "PyModule_GetDict() failed" << std::endl;
    exit(-1);
  }
  PyObject* pClass = PyDict_GetItemString(pDict,"myclass");

  if (!PyCallable_Check(pClass)) {
    std::cout << "PyCallable_Check(pClass) failed" << std::endl;
    exit(-1);
  }
  PyObject* pInstance = PyObject_CallObject(pClass, NULL);

  pName = PyUnicode_FromString("mymethod");
  PyObject* result=PyObject_CallMethodObjArgs(pInstance, pName,NULL);
  if (!result){
    std::cout << "Error calling PyObject_CallMethodObjArgs" << std::endl;
    PyErr_Print();
    exit(-1);
  }
  Py_DECREF(pName);
  Py_DECREF(pModule);
  Py_Finalize();

  return 0;
}

Это построено с

g++ -g -O0 -o test_embed_python_static test_static.cpp -I/usr/include/python3.7m -lpython3.7m

Мой Python модуль:

mymodule.py

import numpy as np

class myclass:
  def __init__(self):
    self.something="asdf"
    self.a=np.zeros( (5),dtype=float)
  def mymethod(self):
    print("Called myclass.mymethod()")
    print('%d'%(self.a.size))

Этот Python модуль находится в локальном каталог и я устанавливаю переменную среды export PYTHONPATH=.

Вывод:

user@host:~/test_embed_python$ ./test_embed_python_static
Called myclass.mymethod()
5

Вот пример, который использует библиотеки Dynami c:

test_dynami c. cpp

#include <dlfcn.h>
typedef void MyCppFunctionHandle();
int main(){
  void* soHandle = dlopen("/home/user/lib/libtestlib.so", RTLD_NOW );
  MyCppFunctionHandle* functionHandle=(MyCppFunctionHandle*)dlsym(soHandle,"MyCppFunction");
  functionHandle();
  return 0;
}

test_lib. cpp

#include <iostream>
#include <Python.h>

extern "C"
void MyCppFunction(){

  Py_Initialize();

  PyObject* pName = PyUnicode_FromString("mymodule");
  PyObject* pModule = PyImport_Import(pName);
  if (!pModule){
    std::cout << "PyImport_Import() failed for " << "mymodule" << std::endl;
    PyErr_Print();
    exit(-1);
  }

  PyObject* pDict = PyModule_GetDict(pModule);
  if (!pDict){
    std::cout << "PyModule_GetDict() failed" << std::endl;
    exit(-1);
  }

  PyObject* pClass = PyDict_GetItemString(pDict,"myclass");

  if (!PyCallable_Check(pClass)) {
    std::cout << "PyCallable_Check(pClass) failed" << std::endl;
    exit(-1);
  }
  PyObject* pInstance = PyObject_CallObject(pClass, NULL);

  pName = PyUnicode_FromString("mymethod");
  PyObject* result=PyObject_CallMethodObjArgs(pInstance, pName,NULL);
  if (!result){
    std::cout << "Error calling PyObject_CallMethodObjArgs" << std::endl;
    PyErr_Print();
    exit(-1);
  }
  Py_DECREF(pName);
  Py_DECREF(pModule);
  Py_Finalize();
}

Сборка библиотеки и основного исполняемого файла:

g++ -g -O0 -fPIC -shared -o libtestlib.so test_lib.cpp -I/usr/include/python3.7m -lpython3.7m
mv libtestlib.so ~/lib/
g++ -g -O0 -o test_embed_python_dynamic test_dynamic.cpp -ldl

Использование того же mymodule.py как указано выше и PYTHONPATH снова установлено на ..

Вывод:

user@host:~/test_embed_python$ ./test_embed_python_dynamic 
Dynamic library opened successfully
Function loaded successfully
Calling function
PyImport_Import() failed for mymodule
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/numpy/core/__init__.py", line 40, in <module>
    from . import multiarray
  File "/usr/lib/python3/dist-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/usr/lib/python3/dist-packages/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ImportError: /usr/lib/python3/dist-packages/numpy/core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PyExc_ImportError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/code/test/test_embed_python/mymodule.py", line 1, in <module>
    import numpy as np
  File "/usr/lib/python3/dist-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/usr/lib/python3/dist-packages/numpy/core/__init__.py", line 71, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
Here is how to proceed:
- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.
- If you are simply trying to use the numpy version that you have installed:
  your installation is broken - please reinstall numpy.
- If you have already reinstalled and that did not fix the problem, then:
  1. Check that you are using the Python you expect (you're using /usr/bin/python3),
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy versions you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

     Note: this error has many possible causes, so please don't comment on
     an existing issue about this - open a new one instead.

Original error was: /usr/lib/python3/dist-packages/numpy/core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PyExc_ImportError

Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ImportError: /usr/lib/python3/dist-packages/apt_pkg.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PyExc_ValueError

Original exception was:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/numpy/core/__init__.py", line 40, in <module>
    from . import multiarray
  File "/usr/lib/python3/dist-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/usr/lib/python3/dist-packages/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ImportError: /usr/lib/python3/dist-packages/numpy/core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PyExc_ImportError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/test_embed_python/mymodule.py", line 1, in <module>
    import numpy as np
  File "/usr/lib/python3/dist-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/usr/lib/python3/dist-packages/numpy/core/__init__.py", line 71, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
Here is how to proceed:
- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.
- If you are simply trying to use the numpy version that you have installed:
  your installation is broken - please reinstall numpy.
- If you have already reinstalled and that did not fix the problem, then:
  1. Check that you are using the Python you expect (you're using /usr/bin/python3),
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy versions you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log


     Note: this error has many possible causes, so please don't comment on
     an existing issue about this - open a new one instead.

Original error was: /usr/lib/python3/dist-packages/numpy/core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PyExc_ImportError

Этот вопрос на Numpy Github выглядит аналогично, но фактическое решение никогда не было размещено:

https://github.com/numpy/numpy/issues/14480

1 Ответ

0 голосов
/ 18 марта 2020

Я наткнулся на решение здесь . Мне все еще неясно, в чем проблема и почему это решает ее, но добавление строки

void*const libpython_handle = dlopen("libpython3.7m.so", RTLD_LAZY | RTLD_GLOBAL);

перед Py_Initialize(); в моем test_lib.cpp устраняет проблему.

Добро пожаловать на сайт PullRequest, где вы можете задавать вопросы и получать ответы от других членов сообщества.
...