InvalidArgumentError: невозможно назначить устройство для операции replica_0 / lambda_1 / Shape - PullRequest
Новая
       91

InvalidArgumentError: невозможно назначить устройство для операции replica_0 / lambda_1 / Shape

0 голосов
/ 19 июня 2020

Я тестирую Yolo-v3 (https://github.com/experiencor/keras-yolo3) с tensorflow-gpu 1.15 и keras 2.3.1. Процесс обучения запускается: 

runfile("train.py",'-c config.json')

Вот распечатанные сообщения: 

Using TensorFlow backend.
WARNING:tensorflow:From train.py:40: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

valid_annot_folder not exists. Spliting the trainining set.
Seen labels:    {'kangaroo': 266}

Given labels:   ['kangaroo']

Training on:    ['kangaroo']

WARNING:tensorflow:From C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\tensorflow_core\python\ops\resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
.....


Loading pretrained weights.

C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\keras\callbacks\callbacks.py:998: UserWarning: `epsilon` argument is deprecated and will be removed, use `min_delta` instead.
  warnings.warn('`epsilon` argument is deprecated and '
Traceback (most recent call last):

  File "C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\tensorflow_core\python\client\session.py", line 1365, in _do_call
    return fn(*args)

  File "C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\tensorflow_core\python\client\session.py", line 1348, in _run_fn
    self._extend_graph()

  File "C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\tensorflow_core\python\client\session.py", line 1388, in _extend_graph
    tf_session.ExtendSession(self._session)

InvalidArgumentError: Cannot assign a device for operation replica_0/lambda_1/Shape: {{node replica_0/lambda_1/Shape}} was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0 ]. Make sure the device specification refers to a valid device.
     [[replica_0/lambda_1/Shape]]


During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "train.py", line 305, in <module>
    _main_(args)

  File "train.py", line 282, in _main_
    max_queue_size   = 8

  File "C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)

  File "C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\keras\engine\training.py", line 1732, in fit_generator
    initial_epoch=initial_epoch)

  File "C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\keras\engine\training_generator.py", line 42, in fit_generator
    model._make_train_function()

  File "C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\keras\engine\training.py", line 333, in _make_train_function
    **self._function_kwargs)

  File "C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\keras\backend\tensorflow_backend.py", line 3006, in function
    v1_variable_initialization()

  File "C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\keras\backend\tensorflow_backend.py", line 420, in v1_variable_initialization
    session = get_session()

  File "C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\keras\backend\tensorflow_backend.py", line 385, in get_session
    return tf_keras_backend.get_session()

  File "C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\tensorflow_core\python\keras\backend.py", line 486, in get_session
    _initialize_variables(session)

  File "C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\tensorflow_core\python\keras\backend.py", line 903, in _initialize_variables
    [variables_module.is_variable_initialized(v) for v in candidate_vars])

  File "C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\tensorflow_core\python\client\session.py", line 956, in run
    run_metadata_ptr)

  File "C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\tensorflow_core\python\client\session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)

  File "C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\tensorflow_core\python\client\session.py", line 1359, in _do_run
    run_metadata)

  File "C:\Users\Dy\Anaconda3\envs\tf1x\lib\site-packages\tensorflow_core\python\client\session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)

InvalidArgumentError: Невозможно назначить устройство для работы replica_0 / lambda_1 / Shape: node replica_0 / lambda_1 / Форма (определенная в C: \ Users \ Dy \ Anaconda3 \ envs \ tf1x \ lib \ site-packages \ tensorflow_core \ python \ framework \ ops.py: 1748) была явно назначена / device: GPU: 0, но доступна устройства: [/ job: localhost / replica: 0 / task: 0 / device: CPU: 0]. Убедитесь, что в спецификации устройства указано действительное устройство. [[replica_0 / lambda_1 / Shape]]

Я не понимаю, что вызвало InvalidArgumentError. Мой tensoflow-gpu установлен неправильно? Или есть какой-то конфликт при развертывании gpu?

...