Мне нравится читать все через гитхаб. Но теперь мне нужно решить проблему, которую я еще не нашел.
Я запускаю
- Visual Studio 2019 на
- Win 10 64 бит
- python 3.7 64-битная
- Tensorflow 2.3.0 только на GPU
- CUDA V10.1.243 с CUDNN
- GPU: GTX-1080Ti
Я тренирую сверточные сети. Во время работы заметил кое-что: инициализация происходит дважды. Я не думаю, что это предназначено для этого, и надеюсь, что это root другой моей проблемы, о которой я не буду здесь упоминать. Шаг за шагом.
Так как же инициализировать CUDA только один раз? Вот информация:
2020-08-05 13:16:07.603055: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-05 13:16:10.323190: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2020-08-05 13:16:10.347449: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:02:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 451.17GiB/s
2020-08-05 13:16:10.347743: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-05 13:16:10.352223: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-08-05 13:16:10.356249: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-08-05 13:16:10.357740: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-08-05 13:16:10.362188: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-08-05 13:16:10.364747: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-08-05 13:16:10.373196: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-08-05 13:16:10.374251: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-08-05 13:16:10.374928: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-08-05 13:16:10.383795: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1d28de782a0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-08-05 13:16:10.383987: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-08-05 13:16:10.384768: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:02:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 451.17GiB/s
2020-08-05 13:16:10.385117: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-05 13:16:10.385334: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-08-05 13:16:10.385466: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-08-05 13:16:10.385595: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-08-05 13:16:10.385758: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-08-05 13:16:10.385930: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-08-05 13:16:10.386211: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-08-05 13:16:10.386962: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-08-05 13:16:11.005566: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-05 13:16:11.005721: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2020-08-05 13:16:11.005810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2020-08-05 13:16:11.006873: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 8678 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
2020-08-05 13:16:11.011234: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1d2b7583c70 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-08-05 13:16:11.011462: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1
Model: "functional_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 512, 512, 1)] 0
_________________________________________________________________
conv2d (Conv2D) (None, 512, 512, 64) 640
_________________________________________________________________
conv2d_1 (Conv2D) (None, 512, 512, 64) 36928
=================================================================
Total params: 37,568
Trainable params: 37,568
Non-trainable params: 0
_________________________________________________________________
2020-08-05 13:16:13.133969: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-08-05 13:16:13.883702: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.
2020-08-05 13:16:13.920654: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
1/1 [==============================] - ETA: 0s - loss: 7.8599 - accuracy: 9.1553e-05
1/1 [==============================] - 0s 280ms/step - loss: 7.8599 - accuracy: 9.1553e-05 - val_loss: 7.7643 - val_accuracy: 8.7738e-05
========================> END: 05.08.2020-13:16:14
Der Thread 'MainThread' (0x1) hat mit Code 0 (0x0) geendet.
Das Programm "python.exe" wurde mit Code 0 (0x0) beendet.
Как видите, данные .dll инициализируются несколько раз.
Я использовал код: main.py
from model import *
import scipy.misc
import numpy as np
import datetime
ascent = scipy.misc.ascent()
ascent = np.reshape(ascent,ascent.shape+(1,))
ascent = np.reshape(ascent,(1,)+ascent.shape)
mask = np.zeros(ascent.shape)
mask[0:10,50:150]=1
model = unet(input_size = (512,512,1))
history = model.fit(ascent, mask, steps_per_epoch=1,epochs=1, validation_data = (ascent, mask), validation_steps =1)
print('========================> END: ', datetime.datetime.n
и model.py
import numpy as np
import os
import skimage.io as io
import skimage.transform as trans
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.models import *
from tensorflow.keras.layers import *
from tensorflow.keras.optimizers import *
def unet(input_size,pretrained_weights = None):
inputs = Input(input_size)
conv1 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(inputs)
conv2 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv1)
model = Model(inputs = inputs, outputs = conv2)
model.compile(optimizer = Adam(lr = 1e-4), loss = 'binary_crossentropy', metrics = ['accuracy'])
#1e-4
model.summary()
return model
У вас есть идеи, что я могу с этим сделать?
С уважением!
РЕДАКТИРОВАТЬ:
Я пытался запустить код с помощью python -v, как упоминалось в AKX. В результате получаю огромный лог. Это слишком большое, чтобы размещать здесь. Но все же вот ссылка на файл: https://www.dropbox.com/s/m8vav1tnewhp41p/issueCUDAinitialize.txt?dl=0
Часть, куда загружается CUDA, следующая:
...
# C:\Users\DGrollmisch\.conda\envs\KI-Project\lib\site-packages\tensorflow\keras\utils\__pycache__\__init__.cpython-37.pyc matches C:\Users\DGrollmisch\.conda\envs\KI-Project\lib\site-packages\tensorflow\keras\utils\__init__.py
# code object from 'C:\\Users\\DGrollmisch\\.conda\\envs\\KI-Project\\lib\\site-packages\\tensorflow\\keras\\utils\\__pycache__\\__init__.cpython-37.pyc'
import 'tensorflow.keras.utils' # <_frozen_importlib_external.SourceFileLoader object at 0x0000013C4AB0DEC8>
# C:\Users\DGrollmisch\.conda\envs\KI-Project\lib\site-packages\tensorflow\keras\wrappers\__pycache__\__init__.cpython-37.pyc matches C:\Users\DGrollmisch\.conda\envs\KI-Project\lib\site-packages\tensorflow\keras\wrappers\__init__.py
# code object from 'C:\\Users\\DGrollmisch\\.conda\\envs\\KI-Project\\lib\\site-packages\\tensorflow\\keras\\wrappers\\__pycache__\\__init__.cpython-37.pyc'
# C:\Users\DGrollmisch\.conda\envs\KI-Project\lib\site-packages\tensorflow\keras\wrappers\scikit_learn\__pycache__\__init__.cpython-37.pyc matches C:\Users\DGrollmisch\.conda\envs\KI-Project\lib\site-packages\tensorflow\keras\wrappers\scikit_learn\__init__.py
# code object from 'C:\\Users\\DGrollmisch\\.conda\\envs\\KI-Project\\lib\\site-packages\\tensorflow\\keras\\wrappers\\scikit_learn\\__pycache__\\__init__.cpython-37.pyc'
import 'tensorflow.keras.wrappers.scikit_learn' # <_frozen_importlib_external.SourceFileLoader object at 0x0000013C4AB11B48>
import 'tensorflow.keras.wrappers' # <_frozen_importlib_external.SourceFileLoader object at 0x0000013C4AB11708>
import 'tensorflow.keras' # <_frozen_importlib_external.SourceFileLoader object at 0x0000013C4AACE848>
import 'tensorflow' # <_frozen_importlib_external.SourceFileLoader object at 0x0000013C430F65C8>
import 'model' # <_frozen_importlib_external.SourceFileLoader object at 0x0000013C2926FF88>
# destroy tensorflow.python.keras.engine.base_layer
# C:\Users\DGrollmisch\.conda\envs\KI-Project\lib\site-packages\tensorflow\python\keras\engine\__pycache__\base_layer_v1.cpython-37.pyc matches C:\Users\DGrollmisch\.conda\envs\KI-Project\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py
# code object from 'C:\\Users\\DGrollmisch\\.conda\\envs\\KI-Project\\lib\\site-packages\\tensorflow\\python\\keras\\engine\\__pycache__\\base_layer_v1.cpython-37.pyc'
import 'tensorflow.python.keras.engine.base_layer_v1' # <_frozen_importlib_external.SourceFileLoader object at 0x0000013C4AB15088>
# destroy tensorflow.python.keras.engine.base_layer_v1
# destroy tensorflow.python.distribute.distribute_lib
2020-08-05 14:45:26.141350: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2020-08-05 14:45:26.161125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:02:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 451.17GiB/s
2020-08-05 14:45:26.161413: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-05 14:45:26.167517: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-08-05 14:45:26.171373: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-08-05 14:45:26.172995: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-08-05 14:45:26.177250: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-08-05 14:45:26.179778: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-08-05 14:45:26.188220: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-08-05 14:45:26.188956: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-08-05 14:45:26.189772: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-08-05 14:45:26.199148: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x13c49ff9c00 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-08-05 14:45:26.199384: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-08-05 14:45:26.200115: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:02:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 451.17GiB/s
2020-08-05 14:45:26.200572: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-05 14:45:26.200937: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-08-05 14:45:26.201175: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-08-05 14:45:26.201358: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-08-05 14:45:26.201550: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-08-05 14:45:26.201815: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-08-05 14:45:26.202166: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-08-05 14:45:26.202942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-08-05 14:45:26.815334: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-05 14:45:26.815493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2020-08-05 14:45:26.815590: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2020-08-05 14:45:26.816590: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 8678 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
2020-08-05 14:45:26.819910: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x13c67f4f600 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-08-05 14:45:26.820091: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1
# destroy tensorflow.python.framework.constant_op
# destroy tensorflow.python.distribute.distribution_strategy_context
# destroy tensorflow.python.keras.engine.training
# destroy tensorflow.python.keras.engine.training_v1
Model: "functional_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 512, 512, 1)] 0
_________________________________________________________________
conv2d (Conv2D) (None, 512, 512, 64) 640
_________________________________________________________________
conv2d_1 (Conv2D) (None, 512, 512, 64) 36928
=================================================================
Total params: 37,568
Trainable params: 37,568
Non-trainable params: 0
_________________________________________________________________
# destroy tensorflow.python.autograph.core.ag_ctx
# destroy tensorflow.python.autograph.core.ag_ctx
# destroy tensorflow.python.autograph.impl.api
# destroy tensorflow.python.framework.ops
# destroy tensorflow.python.framework.tensor_spec
...
Здесь файл requirements.txt с установленными пакетами в моем окружении:
absl-py==0.9.0
astor==0.8.1
astropy==4.0.1.post1
astunparse==1.6.3
atomicwrites==1.4.0
attrs==19.3.0
backcall==0.2.0
bleach==3.1.5
cachetools==4.1.0
certifi==2020.4.5.2
chardet==3.0.4
colorama==0.4.3
cycler==0.10.0
decorator==4.4.2
defusedxml==0.6.0
entrypoints==0.3
gast==0.3.3
google-auth==1.17.2
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.29.0
h5py==2.10.0
idna==2.9
imageio==2.8.0
importlib-metadata==1.6.1
install==1.3.3
ipykernel==5.3.0
ipython==7.15.0
ipython-genutils==0.2.0
jedi==0.17.0
Jinja2==2.11.2
jsonschema==3.2.0
jupyter-client==6.1.3
jupyter-core==4.6.3
Keras-Preprocessing==1.1.2
kiwisolver==1.2.0
libtiff==0.4.2
Markdown==3.2.2
MarkupSafe==1.1.1
matplotlib==3.2.2
memory-profiler==0.57.0
mistune==0.8.4
more-itertools==8.4.0
nbconvert==5.6.1
nbformat==5.0.7
networkx==2.4
noise==1.2.2
notebook==6.0.3
numpy==1.18.5
oauthlib==3.1.0
olefile==0.46
opencv-python==4.2.0.34
opensimplex==0.3
opt-einsum==3.2.1
packaging==20.4
pandocfilters==1.4.2
parso==0.7.0
pickleshare==0.7.5
Pillow==7.1.2
pip==20.2.1
pluggy==0.13.1
prometheus-client==0.8.0
prompt-toolkit==3.0.5
protobuf==3.12.2
psutil==5.7.2
py==1.8.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
Pygments==2.6.1
pyparsing==2.4.7
pyrsistent==0.16.0
pytest==5.4.2
python-dateutil==2.8.1
PyWavelets==1.1.1
pywin32==227
pywinpty==0.5.7
PyYAML==5.3.1
pyzmq==18.1.1
regions==0.4
requests==2.24.0
requests-oauthlib==1.3.0
rsa==4.6
scikit-image==0.17.2
scipy==1.4.1
Send2Trash==1.5.0
setuptools==47.3.1
six==1.15.0
tensorboard==2.3.0
tensorboard-plugin-wit==1.7.0
tensorflow-gpu==2.3.0
tensorflow-gpu-estimator==2.3.0
termcolor==1.1.0
terminado==0.8.3
testpath==0.4.4
tifffile==2020.6.3
tornado==6.0.4
traitlets==4.3.3
urllib3==1.25.9
wcwidth==0.2.4
webencodings==0.5.1
Werkzeug==1.0.1
wheel==0.34.2
wincertstore==0.2
wrapt==1.12.1
zipp==3.1.0