TensorFlow 2.1 с графическим процессором NVidia, возвращающий предупреждения и ошибки: отсутствующие библиотеки, NUMA, операция свертки - PullRequest
Новая
       56

TensorFlow 2.1 с графическим процессором NVidia, возвращающий предупреждения и ошибки: отсутствующие библиотеки, NUMA, операция свертки

0 голосов
/ 08 мая 2020

Я пытаюсь обучить нейронную сеть на tensorflow 2.1.0. Я установил все необходимое программное обеспечение для настройки графического процессора NVidia RTX 2070. Фактически, когда я набираю: tf.test.is_gpu_available(), я получаю True.

Однако именно это начало происходить со мной, когда я import tensorflow as tf, в начале каждого запуска. Это появляется в терминале: 

2020-05-08 10:07:48.506283: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-05-08 10:07:48.506523: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvrtc.so.10.2: cannot open shared object file: No such file or directory
2020-05-08 10:07:48.506534: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2020-05-08 10:07:49.047809: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-08 10:07:49.084978: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-08 10:07:49.085264: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce RTX 2070 computeCapability: 7.5
coreClock: 1.44GHz coreCount: 36 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s
2020-05-08 10:07:49.085420: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-08 10:07:49.085476: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-08 10:07:49.086628: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-08 10:07:49.086807: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-08 10:07:49.087975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-08 10:07:49.088620: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-08 10:07:49.088643: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-08 10:07:49.088700: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-08 10:07:49.088997: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-08 10:07:49.089251: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0

И позже, когда начинается фактическое обучение модели, я получаю: 

2020-05-08 10:07:49.235606: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-05-08 10:07:49.258082: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2599990000 Hz
2020-05-08 10:07:49.258706: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5c2fe60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-08 10:07:49.258733: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-05-08 10:07:49.330241: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-08 10:07:49.330585: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5c1e240 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-08 10:07:49.330600: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce RTX 2070, Compute Capability 7.5
2020-05-08 10:07:49.330749: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-08 10:07:49.331031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce RTX 2070 computeCapability: 7.5
coreClock: 1.44GHz coreCount: 36 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s
2020-05-08 10:07:49.331057: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-08 10:07:49.331065: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-08 10:07:49.331072: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-08 10:07:49.331100: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-08 10:07:49.331108: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-08 10:07:49.331116: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-08 10:07:49.331135: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-08 10:07:49.331185: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-08 10:07:49.331517: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-08 10:07:49.331778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-05-08 10:07:49.331799: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-08 10:07:49.332395: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-08 10:07:49.332404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]      0 
2020-05-08 10:07:49.332408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0:   N 
2020-05-08 10:07:49.332499: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-08 10:07:49.332793: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-08 10:07:49.333078: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6381 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070, pci bus id: 0000:01:00.0, compute capability: 7.5)

и 

2020-05-08 10:08:04.498028: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-08 10:08:04.798897: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-08 10:08:05.159827: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-05-08 10:08:05.161453: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-05-08 10:08:05.161572: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
     [[{{node model/conv1d/conv1d}}]]
2020-05-08 10:08:05.163161: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-05-08 10:08:05.163198: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cudnn_rnn_ops.cc:1510 : Unknown: Fail to find the dnn implementation.
2020-05-08 10:08:05.163233: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Unknown: Fail to find the dnn implementation.
     [[{{node CudnnRNN}}]]

и 

tensorflow.python.framework.errors_impl.UnknownError:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
     [[node model/conv1d/conv1d (defined at home/ivan/Documents/ML/projects/rnn/wtf_imputation/GAN-RNN_Timeseries-imputation/train.py:71) ]] [Op:__inference_train_on_batch_5414]

Failed to get convolution algorithm - это то, что я решил в прошлом, добавив этот блок в начало моего тренировочного сценария: 

import tensorflow as tf
# Solves Convolution CuDNN error
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    try:
        for gpu in gpus:
            tf.config.experimental.set_memory_growth(gpu, True)
    except RuntimeError as e:
        print(e)

Но на этот раз он не работает, и я не совсем понимаю, почему.

РЕДАКТИРОВАТЬ:

Несмотря на то, что там написано, что у меня CUDA 10.2, я фактически установил версию 10.1 по запросу TensorFlow. Фактически, когда я че c nvcc --version, я получаю: 

[...]
Cuda compilation tools, release 10.1, V10.1.243

Итак, у меня версия 10.1. Я не понимаю, в чем проблема.

1 Ответ

1 голос
/ 08 мая 2020

1) Невозможно открыть некоторые библиотеки TensorRT.

Вы либо не установили библиотеки TensorRT (они не зависят от Tensorflow и CUDA и предлагают некоторые c и дополнительные возможности ускорения). пока проигнорируйте это, посмотрите, как установить библиотеки (на странице установки TF ) для получения дополнительной информации о том, как их установить.

2) Не удалось создать дескриптор cudnn: CUDNN_STATUS_INTERNAL_ERROR

Обычно это вызвано отсутствием установленного CuDNN или неправильной версией. Поскольку написано Successfully opened dynamic library libcudnn.so.7, я склоняюсь ко второму варианту. Убедитесь, что версия, которую вы установили, соответствует версии, требуемой Tensorflow (которая может быть более старой, чем последняя версия, доступная на веб-сайте NVIDIA).

В качестве примечания, из ваших журналов кажется, что вы у вас установлен CUDA 10.2. Tensorflow требует версии 10.1, так что это может быть еще одним источником проблем. В этом случае вы можете установить версию 10.1 вместе с версией 10.2 в своей системе или удалить 10.2 и сэкономить место.

Изменить : 10.2 в журналах относится к библиотекам TensorRT, в остальных журналах перечислены библиотеки с версией 10.1, поэтому примечание, вероятно, неверно.

...