Ошибка графического процессора Tensorflow CUDA_ERROR_OUT_OF_MEMORY: недостаточно памяти - PullRequest
0 голосов
/ 08 января 2019

Я новичок в tenorflow и у меня возникли проблемы с запуском в GPU, в CPU все в порядке.

Когда я запускаю следующую команду, чтобы проверить установку тензор потока:

python -c "import tensorflow as tf; tf.enable_eager_execution(); print(tf.reduce_sum(tf.random_normal([1000, 1000])))" 

Я получаю эту ошибку:

    2019-01-08 18:49:51.551078: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/myUsername/.local/lib/python3.6/site-packages/tensorflow/python/ops/random_ops.py", line 73, in random_normal
    shape_tensor = _ShapeTensor(shape)
  File "/home/myUsername/.local/lib/python3.6/site-packages/tensorflow/python/ops/random_ops.py", line 44, in _ShapeTensor
    return ops.convert_to_tensor(shape, dtype=dtype, name="shape")
  File "/home/myUsername/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1050, in convert_to_tensor
    as_ref=False)
  File "/home/myUsername/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1146, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/myUsername/.local/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 229, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/home/myUsername/.local/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 179, in constant
    t = convert_to_eager_tensor(value, ctx, dtype)
  File "/home/myUsername/.local/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 99, in convert_to_eager_tensor
    handle = ctx._handle  # pylint: disable=protected-access
  File "/home/myUsername/.local/lib/python3.6/site-packages/tensorflow/python/eager/context.py", line 319, in _handle
    self._initialize_handle_and_devices()
  File "/home/myUsername/.local/lib/python3.6/site-packages/tensorflow/python/eager/context.py", line 267, in _initialize_handle_and_devices
    self._context_handle = pywrap_tensorflow.TFE_NewContext(opts)
tensorflow.python.framework.errors_impl.InternalError: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY: out of memory; total memory reported: 12788498432

Также со следующим примером:

import tensorflow as tf
mnist = tf.keras.datasets.mnist

(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(512, activation=tf.nn.relu),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test)

Я получаю эту ошибку:

2019-01-08 18:53:07.267303: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
Traceback (most recent call last):
  File "test_keras.py", line 17, in <module>
    model.fit(x_train, y_train, epochs=5)
  File "/home/myUsername/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1639, in fit
    validation_steps=validation_steps)
  File "/home/myUsername/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 215, in fit_loop
    outs = f(ins_batch)
  File "/home/myUsername/.local/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 2947, in __call__
    session = get_session()
  File "/home/myUsername/.local/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 465, in get_session
    _SESSION = session_module.Session(config=get_default_session_config())
  File "/home/myUsername/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1551, in __init__
    super(Session, self).__init__(target, graph, config=config)
  File "/home/myUsername/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 676, in __init__
    self._session = tf_session.TF_NewSessionRef(self._graph._c_graph, opts)
tensorflow.python.framework.errors_impl.InternalError: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY: out of memory; total memory reported: 12788498432

Любая инструкция о том, как решить эту проблему ????

Описание моей системы:

python3 -V

Python 3.6.7

nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176

NVIDIA-SMI

Tue Jan  8 18:37:03 2019       
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 390.87                 Driver Version: 390.87                    |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |===============================+======================+======================|
    |   0  TITAN Xp            Off  | 00000000:17:00.0 Off |                  N/A |
    | 23%   31C    P8    16W / 250W |  12176MiB / 12196MiB |      0%      Default |
    +-------------------------------+----------------------+----------------------+
    |   1  GeForce GTX 1070    Off  | 00000000:65:00.0  On |                  N/A |
    |  0%   48C    P8    13W / 180W |   7768MiB /  8118MiB |      0%      Default |
    +-------------------------------+----------------------+----------------------+

ВЕРСИЯ CuDNN

 cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2
#define CUDNN_MAJOR 7
#define CUDNN_MINOR 0
#define CUDNN_PATCHLEVEL 5
--
#define CUDNN_VERSION    (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

#include "driver_types.h"

версия tenorflow

python3
Python 3.6.7 (default, Oct 22 2018, 11:32:17) 
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.__version__
'1.12.0'

1 Ответ

0 голосов
/ 09 января 2019

Роберт Кровелла, спасибо за ваш ответ.

Я выполнил шаги, которые вы мне сказали, и у меня та же проблема. Это результат. Как видите, использование памяти очень мало: 2 МБ для Titan XP и 902 МБ для GTX1070.

nvidia-smi

Wed Jan  9 10:56:55 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.87                 Driver Version: 390.87                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN Xp            Off  | 00000000:17:00.0 Off |                  N/A |
| 23%   22C    P8     8W / 250W |      2MiB / 12196MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 1070    Off  | 00000000:65:00.0  On |                  N/A |
|  0%   37C    P8    10W / 180W |    902MiB /  8118MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    1      1355      G   /usr/lib/xorg/Xorg                            40MiB |
|    1      1588      G   /usr/bin/gnome-shell                          81MiB |
|    1      3342      G   /usr/lib/xorg/Xorg                           439MiB |
|    1      3535      G   /usr/bin/gnome-shell                         227MiB |
|    1      8880      G   ...uest-channel-token=15629967551314695332   109MiB |
|    1     26921      G   /usr/bin/nvidia-settings                       0MiB |
+-----------------------------------------------------------------------------+

Когда я установил тензор потока, я следовал этому уроку ссылка с основным отличием, что я установил тензор потока 1.12 в unbuntu 18.10

...