Я работаю над созданием кодовой базы машинного обучения, которая имеет много зависимостей от современных библиотек машинного обучения (sklearn, xgboost, tenorflow, pandas и тому подобное). Мы пытаемся объединить кодовую базу в одну папку, содержащую исполняемый файл и дополнительные двоичные файлы, необходимые для работы (общие библиотеки). Для этого мы пытаемся использовать PyInstaller.
Я могу успешно создать исполняемый файл на MacOS (mojave 10.14.6) и CentOS 7.7. Я использую conda (4.7.12) для управления виртуальными средами и Python 3.6.6. На обеих платформах я могу запустить код из исходного кода (т.е. не из exe, созданного PyInstaller), и все работает нормально. Запуск исполняемого файла на MacOS работает нормально. Однако запуск исполняемого файла в CentOS завершается ошибкой сегмента (неясно, где это происходит на самом деле).
Вот вывод процесса сборки в CentOS:
54 INFO: PyInstaller: 3.5
54 INFO: Python: 3.6.6
55 INFO: Platform: Linux-3.10.0-1062.1.2.el7.x86_64-x86_64-with-centos-7.7.1908-Core
58 INFO: UPX is not available.
59 INFO: Removing temporary files and cleaning cache in /root/.cache/pyinstaller
60 INFO: Extending PYTHONPATH with paths
['/home/jloy/amb-rte', '/home/jloy/amb-rte']
60 INFO: checking Analysis
61 INFO: Building Analysis because Analysis-00.toc is non existent
61 INFO: Initializing module dependency graph...
64 INFO: Initializing module graph hooks...
66 INFO: Analyzing base_library.zip ...
3879 INFO: Analyzing hidden import 'amb_pacman.settings.rte'
4869 INFO: Analyzing hidden import 'amb.settings.rte'
5161 INFO: Processing pre-find module path hook distutils
6708 INFO: Processing pre-safe import module hook six.moves
7724 INFO: Processing pre-safe import module hook setuptools.extern.six.moves
8357 INFO: Processing pre-find module path hook site
8358 INFO: site: retargeting to fake-dir '/root/miniconda3/envs/rte/lib/python3.6/site-packages/PyInstaller/fake-modules'
37973 INFO: Analyzing hidden import 'amb.evolve.neat.core.indexer'
38049 INFO: Analyzing hidden import 'amb_utils.settings.rte'
39648 INFO: Analyzing hidden import 'sklearn.utils._cython_blas'
39648 INFO: Analyzing hidden import 'sklearn.neighbors.typedefs'
39648 INFO: Analyzing hidden import 'sklearn.neighbors.quad_tree'
39649 INFO: Analyzing hidden import 'sklearn.tree._utils'
39734 INFO: Analyzing hidden import 'multiprocessing.forking'
39734 ERROR: Hidden import 'multiprocessing.forking' not found
39734 INFO: Analyzing hidden import 'multiprocessing.freeze_support'
39734 ERROR: Hidden import 'multiprocessing.freeze_support' not found
39734 INFO: Analyzing hidden import 'numpy.type'
39735 ERROR: Hidden import 'numpy.type' not found
39735 INFO: running Analysis Analysis-00.toc
39762 INFO: Caching module hooks...
39768 INFO: Analyzing /home/jloy/amb-rte/runtime_engine.py
51355 INFO: Processing pre-safe import module hook win32com
68391 INFO: Processing pre-safe import module hook urllib3.packages.six.moves
101910 INFO: Loading module hooks...
101911 INFO: Loading module hook "hook-IPython.py"...
101926 INFO: Excluding import 'matplotlib'
101941 INFO: Removing import of matplotlib._pylab_helpers from module IPython.core.pylabtools
101941 INFO: Removing import of matplotlib.pyplot from module IPython.core.pylabtools
101941 INFO: Removing import of matplotlib.figure from module IPython.core.pylabtools
101941 INFO: Removing import of matplotlib from module IPython.core.pylabtools
101943 INFO: Excluding import 'PySide'
101957 INFO: Excluding import 'PyQt5'
101972 INFO: Excluding import 'PyQt4'
101986 INFO: Excluding import 'gtk'
102000 INFO: Loading module hook "hook-PIL.Image.py"...
102458 INFO: Loading module hook "hook-PIL.SpiderImagePlugin.py"...
102483 INFO: Import to be excluded not found: 'FixTk'
102484 INFO: Excluding import 'tkinter'
102496 INFO: Loading module hook "hook-PIL.py"...
102506 INFO: Excluding import 'PySide'
102521 INFO: Removing import of PySide from module PIL.ImageQt
102522 INFO: Excluding import 'PyQt5'
102534 INFO: Removing import of PyQt5.QtCore from module PIL.ImageQt
102534 INFO: Removing import of PyQt5.QtGui from module PIL.ImageQt
102535 INFO: Excluding import 'PyQt4'
102547 INFO: Removing import of PyQt4 from module PIL.ImageQt
102548 INFO: Import to be excluded not found: 'FixTk'
102548 INFO: Excluding import 'tkinter'
102559 INFO: Removing import of tkinter from module PIL.ImageTk
102559 INFO: Loading module hook "hook-PyQt5.QtCore.py"...
102708 INFO: Loading module hook "hook-PyQt5.QtGui.py"...
102832 INFO: Loading module hook "hook-PyQt5.QtWidgets.py"...
103067 INFO: Loading module hook "hook-PyQt5.py"...
103489 WARNING: Hidden import "PyQt5.sip" not found!
103489 INFO: Loading module hook "hook-_tkinter.py"...
103656 INFO: checking Tree
103657 INFO: Building Tree because Tree-00.toc is non existent
103657 INFO: Building Tree Tree-00.toc
103665 INFO: checking Tree
103666 INFO: Building Tree because Tree-01.toc is non existent
103666 INFO: Building Tree Tree-01.toc
103670 INFO: Loading module hook "hook-certifi.py"...
103672 INFO: Loading module hook "hook-cryptography.py"...
103978 INFO: Loading module hook "hook-distutils.py"...
103979 INFO: Loading module hook "hook-encodings.py"...
104077 INFO: Loading module hook "hook-h5py.py"...
104079 INFO: Loading module hook "hook-importlib_metadata.py"...
104082 INFO: Loading module hook "hook-jedi.py"...
104127 INFO: Loading module hook "hook-jinja2.py"...
104157 INFO: Loading module hook "hook-lib2to3.py"...
104159 INFO: Loading module hook "hook-llvmlite.py"...
104161 INFO: Loading module hook "hook-matplotlib.backends.py"...
104933 INFO: Matplotlib backend "GTK": ignored
Gtk* backend requires pygtk to be installed.
105583 INFO: Matplotlib backend "GTKAgg": ignored
Gtk* backend requires pygtk to be installed.
106039 INFO: Matplotlib backend "GTKCairo": ignored
No module named 'gtk'
106663 INFO: Matplotlib backend "GTK3Agg": ignored
The Gtk3 backend requires PyGObject or pgi
107050 INFO: Matplotlib backend "GTK3Cairo": ignored
cairo backend requires that cairocffi or pycairo is installed
107625 INFO: Matplotlib backend "MacOSX": ignored
cannot import name '_macosx'
/root/miniconda3/envs/rte/lib/python3.6/site-packages/IPython/kernel/__init__.py:13: ShimWarning: The `IPython.kernel` package has been deprecated since IPython 4.0.You should import from ipykernel or jupyter_client instead.
"You should import from ipykernel or jupyter_client instead.", ShimWarning)
108462 INFO: Matplotlib backend "nbAgg": ignored
No module named 'ipykernel'
109134 INFO: Matplotlib backend "Qt4Agg": added
109487 INFO: Matplotlib backend "Qt4Cairo": ignored
cairo backend requires that cairocffi or pycairo is installed
110150 INFO: Matplotlib backend "Qt5Agg": added
110519 INFO: Matplotlib backend "Qt5Cairo": ignored
cairo backend requires that cairocffi or pycairo is installed
111089 INFO: Matplotlib backend "TkAgg": added
111452 INFO: Matplotlib backend "TkCairo": ignored
cairo backend requires that cairocffi or pycairo is installed
112115 INFO: Matplotlib backend "WebAgg": added
112741 INFO: Matplotlib backend "WX": ignored
Matplotlib backend_wx and backend_wxagg require wxPython>=2.9
113172 INFO: Matplotlib backend "WXAgg": ignored
No module named 'wx'
113603 INFO: Matplotlib backend "WXCairo": ignored
No module named 'wx'
114279 INFO: Matplotlib backend "agg": added
114704 INFO: Matplotlib backend "cairo": ignored
cairo backend requires that cairocffi or pycairo is installed
115090 INFO: Matplotlib backend "gdk": ignored
No module named 'gobject'
115640 INFO: Matplotlib backend "pdf": added
116288 INFO: Matplotlib backend "pgf": added
116838 INFO: Matplotlib backend "ps": added
117424 INFO: Matplotlib backend "svg": added
117967 INFO: Matplotlib backend "template": added
118562 INFO: Loading module hook "hook-matplotlib.py"...
118991 INFO: Loading module hook "hook-numba.py"...
119014 INFO: Excluding import 'IPython'
119033 INFO: Removing import of IPython.paths from module numba.caching
119033 INFO: Removing import of IPython.utils.path from module numba.caching
119040 INFO: Excluding import 'scipy'
119057 INFO: Removing import of scipy.linalg.cython_blas from module numba.targets.linalg
119057 INFO: Removing import of scipy.linalg.cython_lapack from module numba.targets.linalg
119063 INFO: Loading module hook "hook-numpy.core.py"...
119229 INFO: MKL libraries found when importing numpy. Adding MKL to binaries
119230 INFO: Loading module hook "hook-numpy.py"...
119231 INFO: Loading module hook "hook-pandas.py"...
120598 INFO: Loading module hook "hook-patsy.py"...
120600 INFO: Loading module hook "hook-pkg_resources.py"...
121246 INFO: Loading module hook "hook-pycparser.py"...
121247 INFO: Loading module hook "hook-pydoc.py"...
121247 INFO: Loading module hook "hook-pygments.py"...
123779 INFO: Loading module hook "hook-pytest.py"...
124808 INFO: Loading module hook "hook-pytz.py"...
124826 INFO: Loading module hook "hook-scipy.io.matlab.py"...
124827 INFO: Loading module hook "hook-scipy.linalg.py"...
124827 INFO: Loading module hook "hook-scipy.py"...
124828 INFO: Loading module hook "hook-scipy.sparse.csgraph.py"...
124831 INFO: Loading module hook "hook-scipy.special._ellip_harm_2.py"...
124831 INFO: Loading module hook "hook-scipy.special._ufuncs.py"...
124832 INFO: Loading module hook "hook-setuptools.py"...
125823 INFO: Loading module hook "hook-shelve.py"...
125824 INFO: Loading module hook "hook-sklearn.metrics.cluster.py"...
125851 WARNING: Hidden import "sklearn.utils.sparsetools._graph_validation" not found!
125851 WARNING: Hidden import "sklearn.utils.sparsetools._graph_tools" not found!
125852 INFO: Loading module hook "hook-sqlite3.py"...
125944 INFO: Loading module hook "hook-sysconfig.py"...
125956 INFO: Loading module hook "hook-torch.py"...
125987 INFO: Loading module hook "hook-xml.dom.domreg.py"...
125987 INFO: Loading module hook "hook-xml.etree.cElementTree.py"...
125988 INFO: Loading module hook "hook-xml.py"...
125988 INFO: Loading module hook "hook-xgboost.py"...
126102 INFO: Determining a mapping of distributions to packages...
157007 INFO: Packages required by xgboost:
['numpy', 'scipy']
157271 INFO: checking Tree
157272 INFO: Building Tree because Tree-02.toc is non existent
157272 INFO: Building Tree Tree-02.toc
157273 INFO: checking Tree
157273 INFO: Building Tree because Tree-03.toc is non existent
157273 INFO: Building Tree Tree-03.toc
157273 INFO: checking Tree
157273 INFO: Building Tree because Tree-04.toc is non existent
157273 INFO: Building Tree Tree-04.toc
157274 INFO: checking Tree
157274 INFO: Building Tree because Tree-05.toc is non existent
157274 INFO: Building Tree Tree-05.toc
157275 INFO: checking Tree
157275 INFO: Building Tree because Tree-06.toc is non existent
157275 INFO: Building Tree Tree-06.toc
157275 INFO: checking Tree
157275 INFO: Building Tree because Tree-07.toc is non existent
157275 INFO: Building Tree Tree-07.toc
157276 INFO: checking Tree
157276 INFO: Building Tree because Tree-08.toc is non existent
157276 INFO: Building Tree Tree-08.toc
157277 INFO: checking Tree
157277 INFO: Building Tree because Tree-09.toc is non existent
157277 INFO: Building Tree Tree-09.toc
157277 INFO: checking Tree
157277 INFO: Building Tree because Tree-10.toc is non existent
157277 INFO: Building Tree Tree-10.toc
157278 INFO: checking Tree
157278 INFO: Building Tree because Tree-11.toc is non existent
157279 INFO: Building Tree Tree-11.toc
157279 INFO: checking Tree
157279 INFO: Building Tree because Tree-12.toc is non existent
157279 INFO: Building Tree Tree-12.toc
157279 INFO: checking Tree
157280 INFO: Building Tree because Tree-13.toc is non existent
157280 INFO: Building Tree Tree-13.toc
157280 INFO: checking Tree
157280 INFO: Building Tree because Tree-14.toc is non existent
157280 INFO: Building Tree Tree-14.toc
157281 INFO: checking Tree
157282 INFO: Building Tree because Tree-15.toc is non existent
157282 INFO: Building Tree Tree-15.toc
157282 INFO: checking Tree
157282 INFO: Building Tree because Tree-16.toc is non existent
157282 INFO: Building Tree Tree-16.toc
157284 INFO: checking Tree
157284 INFO: Building Tree because Tree-17.toc is non existent
157284 INFO: Building Tree Tree-17.toc
157285 INFO: checking Tree
157285 INFO: Building Tree because Tree-18.toc is non existent
157285 INFO: Building Tree Tree-18.toc
157307 INFO: checking Tree
157307 INFO: Building Tree because Tree-19.toc is non existent
157307 INFO: Building Tree Tree-19.toc
157315 INFO: checking Tree
157316 INFO: Building Tree because Tree-20.toc is non existent
157316 INFO: Building Tree Tree-20.toc
157316 INFO: Looking for ctypes DLLs
157838 WARNING: library user32 required via ctypes not found
159961 INFO: Analyzing run-time hooks ...
160067 INFO: Including run-time hook 'pyi_rth_traitlets.py'
160073 INFO: Including run-time hook 'pyi_rth__tkinter.py'
160078 INFO: Including run-time hook 'pyi_rth_mplconfig.py'
160082 INFO: Including run-time hook 'pyi_rth_mpldata.py'
160085 INFO: Including run-time hook 'pyi_rth_pyqt5.py'
160088 INFO: Including run-time hook 'pyi_rth_pkgres.py'
160092 INFO: Including run-time hook 'pyi_rth_certifi.py'
160096 INFO: Including run-time hook 'pyi_rth_multiprocessing.py'
160297 INFO: Looking for dynamic libraries
ldd: warning: you do not have execution permission for `/root/miniconda3/envs/rte/lib/python3.6/site-packages/cryptography-2.5-py3.6-linux-x86_64.egg/cryptography/hazmat/bindings/_constant_time.abi3.so'
ldd: warning: you do not have execution permission for `/root/miniconda3/envs/rte/lib/python3.6/site-packages/cryptography-2.5-py3.6-linux-x86_64.egg/cryptography/hazmat/bindings/_openssl.abi3.so'
ldd: warning: you do not have execution permission for `/root/miniconda3/envs/rte/lib/python3.6/site-packages/cryptography-2.5-py3.6-linux-x86_64.egg/cryptography/hazmat/bindings/_padding.abi3.so'
ldd: warning: you do not have execution permission for `/root/miniconda3/envs/rte/lib/python3.6/site-packages/coverage-5.0a8-py3.6-linux-x86_64.egg/coverage/tracer.cpython-36m-x86_64-linux-gnu.so'
/root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/tbbpool.cpython-36m-x86_64-linux-gnu.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by /root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2)
/root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/tbbpool.cpython-36m-x86_64-linux-gnu.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2)
/root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/tbbpool.cpython-36m-x86_64-linux-gnu.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2)
ldd: warning: you do not have execution permission for `/root/miniconda3/envs/rte/plugins/platforms/../../lib/libgcc_s.so.1'
/root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2: /lib64/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by /root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2)
/root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2)
/root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2)
173700 INFO: Looking for eggs
173701 INFO: Using Python library /root/miniconda3/envs/rte/lib/python3.6/site-packages/PIL/../../../libpython3.6m.so.1.0
173911 INFO: Warnings written to /home/jloy/amb-rte/pydist/build/runtime_engine/warn-runtime_engine.txt
174792 INFO: Graph cross-reference written to /home/jloy/amb-rte/pydist/build/runtime_engine/xref-runtime_engine.html
175119 INFO: Appending 'datas' from .spec
175132 INFO: checking PYZ
175132 INFO: Building PYZ because PYZ-00.toc is non existent
175132 INFO: Building PYZ (ZlibArchive) /home/jloy/amb-rte/pydist/build/runtime_engine/PYZ-00.pyz
188154 INFO: Building PYZ (ZlibArchive) /home/jloy/amb-rte/pydist/build/runtime_engine/PYZ-00.pyz completed successfully.
188323 INFO: checking PKG
188324 INFO: Building PKG because PKG-00.toc is non existent
188325 INFO: Building PKG (CArchive) PKG-00.pkg
188418 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
188420 INFO: Bootloader /root/miniconda3/envs/rte/lib/python3.6/site-packages/PyInstaller/bootloader/Linux-64bit/run_d
188420 INFO: checking EXE
188420 INFO: Building EXE because EXE-00.toc is non existent
188420 INFO: Building EXE from EXE-00.toc
188420 INFO: Appending archive to ELF section in EXE /home/jloy/amb-rte/pydist/build/runtime_engine/runtime_engine
188523 INFO: Building EXE from EXE-00.toc completed successfully.
188535 INFO: checking COLLECT
188536 INFO: Building COLLECT because COLLECT-00.toc is non existent
188536 INFO: Building COLLECT COLLECT-00.toc
193501 INFO: Building COLLECT COLLECT-00.toc completed successfully.
Предупреждение lddи сообщения о GLIBCXX не происходят для MacOS. В любом случае, процесс создает исполняемый файл, который можно запустить.
Я очень озадачен тем, как это возможно, я могу выполнить код без упаковки без проблем, и в то же время исполняемый файл завершается с ошибкойнеисправность. Тем более, что я работаю на той же машине, на которой был создан исполняемый файл. То есть я по сути делаю это:
$ pyinstaller mySpec.spec
$ ./dist/myExe
Любая помощь по этому вопросу будет принята с благодарностью. Я был бы рад предоставить дополнительную информацию.
Заранее спасибо.
ОБНОВЛЕНИЕ:
Мне удалось запустить исполняемый файлCentOS 7.7. В итоге возникла проблема, с которой использовалась библиотека BLAS. Когда я заставил conda использовать аромат openblas (вместо MKL), все заработало.
Я смог понять это, попробовав ту же процедуру на CentOS 8. При запуске исполняемого файла я получил более описательныйошибка, а не просто ошибка сегмента. Исполняемый файл все еще падал, но это произошло потому, что он не смог найти символ __kmpc_global_thread_num в libmkl_intel_thread.so. Я начал гуглить и смог найти кого-то с похожей проблемой, который решил ее путем принудительного открытия openblas через MKL.
Тем не менее, остается вопрос: зачем работать из исходного кода, а запускать исполняемый файл нет?