CUDA_ERROR_OUT_OF_MEMORY: недостаточно памяти: для тензорного потока 2.1 - PullRequest
Новая
       6

CUDA_ERROR_OUT_OF_MEMORY: недостаточно памяти: для тензорного потока 2.1

1 голос
/ 23 апреля 2020

Я новичок в tenorflow-gpu, работа на CPU выглядит нормально, но почему-то не получается заставить работать версию GPU. Пожалуйста, дайте мне знать, что мне делать дальше. Большое спасибо!

Я использую Python 3.7.7 с TensorFlow 2.1 и установил его, используя 

conda install tensorflow-gpu

Системные характеристики: 

Intel(R) core(TM) I5-7440HQ CPU @ 2.80 GHZ
RAM: 8GB

Характеристики графического процессора : 

Model: GeForce 930MX
GPU memory: 5.9 GB
Dedicated GPU memory: 2GB
Shared GPU memory: 3.9 GB

NVIDIA-SMI 

 NVIDIA-SMI 445.87       Driver Version: 445.87       CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 930MX      WDDM  | 00000000:02:00.0 Off |                  N/A |
| N/A   49C    P8    N/A /  N/A |     37MiB /  2048MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU                  PID   Type   Process name                  GPU Memory |
|                                                                  Usage      |

Запуск простого обучения набора данных MNIST с размером пакета 32.

Командная строка ноутбука Jupyter: 

2020-04-23 12:43:12.448744: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
    2020-04-23 12:43:18.625257: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
    2020-04-23 12:43:18.863674: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
    pciBusID: 0000:02:00.0 name: GeForce 930MX computeCapability: 5.0
    coreClock: 1.0195GHz coreCount: 3 deviceMemorySize: 2.00GiB deviceMemoryBandwidth: 13.41GiB/s
    2020-04-23 12:43:18.869948: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
    2020-04-23 12:43:18.943177: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
    2020-04-23 12:43:19.004099: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
    2020-04-23 12:43:19.030424: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
    2020-04-23 12:43:19.092306: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
    2020-04-23 12:43:19.139074: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
    2020-04-23 12:43:19.264762: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
    2020-04-23 12:43:19.436399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
    2020-04-23 12:43:19.443444: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
    2020-04-23 12:43:19.455503: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
    pciBusID: 0000:02:00.0 name: GeForce 930MX computeCapability: 5.0
    coreClock: 1.0195GHz coreCount: 3 deviceMemorySize: 2.00GiB deviceMemoryBandwidth: 13.41GiB/s
    2020-04-23 12:43:19.463043: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
    2020-04-23 12:43:19.467340: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
    2020-04-23 12:43:19.470920: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
    2020-04-23 12:43:19.477116: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
    2020-04-23 12:43:19.486208: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
    2020-04-23 12:43:19.494696: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
    2020-04-23 12:43:19.505751: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
    2020-04-23 12:43:19.515014: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
    2020-04-23 12:43:24.165525: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
    2020-04-23 12:43:24.169026: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]      0
    2020-04-23 12:43:24.171068: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0:   N
    2020-04-23 12:43:24.175336: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1377 MB memory) -> physical GPU (device: 0, name: GeForce 930MX, pci bus id: 0000:02:00.0, compute capability: 5.0)
    2020-04-23 12:43:24.219369: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 1.34G (1444337920 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.237697: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 1.21G (1299904256 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.260040: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 1.09G (1169913856 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.284695: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 1004.14M (1052922624 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.306355: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 903.73M (947630336 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.327752: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 813.36M (852867328 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.357554: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 732.02M (767580672 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.384318: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 658.82M (690822656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.406377: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 592.94M (621740544 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.426737: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 533.64M (559566592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    [I 12:43:26.566 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports

Вот моя модель, которую я пытаюсь тренировать. Работает нормально на процессоре. 

model = Sequential()

model.add(Conv2D(256, (3, 3), input_shape=X.shape[1:]))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(256, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors

model.add(Dense(64))
model.add(Activation('relu'))

model.add(Dense(1))
model.add(Activation('sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

model.fit(X, y, batch_size=32, epochs=3, validation_split=0.3)

1 Ответ

0 голосов
/ 25 апреля 2020

Cuda11.0 не совместим с Tensorflow2.1. Пожалуйста, проверьте совместимость здесь .

Версия Python версия Компилятор Инструменты сборки cuDNN CUDA.
tenorflow-2.1.0 2.7, 3.5-3.7 G CC 7.3.1 Bazel 0.27.1 7.6 10.1

Tensorflow 2.1 совместим с Cuda10.1. Таким образом, у вас есть две опции

опция 1

создать среду conda и установить tenorflow-gpu == 2.1 

conda create -n tf_gpu
source activete tf_gpu
Within the virtual environment
conda install tensorflow-gpu=2.1

несколько раз следующая работа 

conda create --name <some_name> tensorflow-gpu=2.1.0 cudatoolkit=10.1 python=3.6

опция 2

Удалите Tensorflow и Cuda11.0, выключите и перезагрузите компьютер, а затем переустановите tenorflow-gpu с помощью приведенных выше команд (для установки на основе conda) или следуйте инструкциям здесь установить с помощью pip.

Добро пожаловать на сайт PullRequest, где вы можете задавать вопросы и получать ответы от других членов сообщества.

Похожие темы

...