Создание пользовательской модели обнаружения объектов - PullRequest
0 голосов
/ 09 мая 2020

Я пытаюсь построить модель обнаружения объектов с моим настраиваемым набором данных, имеющим только 1 класс. При выполнении всех процедур, описанных в руководстве, скрипт вылетает из строя и выдает следующую ошибку:

tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
     [[{{node FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d}}]]
     [[Loss/unstack_1/_10307]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
     [[{{node FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d}}]]
0 successful operations.
0 derived errors ignored.
    tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
     [[{{node FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d}}]]
     [[Loss/unstack_1/_10307]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
     [[{{node FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "model_main.py", line 109, in <module>
    tf.app.run()
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate
    return executor.run()
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 613, in run
    return self.run_local()
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
    saving_listeners=saving_listeners)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 367, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1192, in _train_model_default
    saving_listeners)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1484, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 754, in run
    run_metadata=run_metadata)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1252, in run
    run_metadata=run_metadata)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1353, in run
    raise six.reraise(*original_exc_info)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1338, in run
    return self._sess.run(*args, **kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1411, in run
    run_metadata=run_metadata)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1169, in run
    return self._sess.run(*args, **kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
     [[node FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d (defined at /home/stud/hammadal/custom-model/models/research/slim/nets/inception_v2.py:129) ]]
     [[Loss/unstack_1/_10307]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
     [[node FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d (defined at /home/stud/hammadal/custom-model/models/research/slim/nets/inception_v2.py:129) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d':
  File "model_main.py", line 109, in <module>
    tf.app.run()
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate
    return executor.run()
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 613, in run
    return self.run_local()
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
    saving_listeners=saving_listeners)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 367, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1188, in _train_model_default
    features, labels, ModeKeys.TRAIN, self.config)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1146, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/home/stud/hammadal/custom-model/models/research/object_detection/model_lib.py", line 308, in model_fn
    features[fields.InputDataFields.true_image_shape])
  File "/home/stud/hammadal/custom-model/models/research/object_detection/meta_architectures/ssd_meta_arch.py", line 600, in predict
    preprocessed_inputs)
  File "/home/stud/hammadal/custom-model/models/research/object_detection/models/ssd_inception_v2_feature_extractor.py", line 130, in extract_features
    scope=scope)
  File "/home/stud/hammadal/custom-model/models/research/slim/nets/inception_v2.py", line 129, in inception_v2_base
    scope=end_point)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 2784, in separable_convolution2d
    outputs = layer.apply(inputs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1479, in apply
    return self.__call__(inputs, *args, **kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 537, in __call__
    outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 634, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 146, in wrapper
    ), args, kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 446, in converted_call
    return _call_unconverted(f, args, kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 253, in _call_unconverted
    return f(*args, **kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/keras/layers/convolutional.py", line 1658, in call
    data_format=conv_utils.convert_data_format(self.data_format, ndim=4))
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py", line 793, in separable_conv2d
    name=name)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 1953, in conv2d
    name=name)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 1071, in conv2d
    data_format=data_format, dilations=dilations, name=name)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "model_main.py", line 109, in <module>
    tf.app.run()
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate
    return executor.run()
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 613, in run
    return self.run_local()
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
    saving_listeners=saving_listeners)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 367, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1192, in _train_model_default
    saving_listeners)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1484, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 754, in run
    run_metadata=run_metadata)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1252, in run
    run_metadata=run_metadata)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1353, in run
    raise six.reraise(*original_exc_info)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1338, in run
    return self._sess.run(*args, **kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1411, in run
    run_metadata=run_metadata)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1169, in run
    return self._sess.run(*args, **kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
     [[node FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d (defined at /home/stud/hammadal/custom-model/models/research/slim/nets/inception_v2.py:129) ]]
     [[Loss/unstack_1/_10307]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
     [[node FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d (defined at /home/stud/hammadal/custom-model/models/research/slim/nets/inception_v2.py:129) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d':
  File "model_main.py", line 109, in <module>
    tf.app.run()
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate
    return executor.run()
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 613, in run
    return self.run_local()
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
    saving_listeners=saving_listeners)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 367, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1188, in _train_model_default
    features, labels, ModeKeys.TRAIN, self.config)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1146, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/home/stud/hammadal/custom-model/models/research/object_detection/model_lib.py", line 308, in model_fn
    features[fields.InputDataFields.true_image_shape])
  File "/home/stud/hammadal/custom-model/models/research/object_detection/meta_architectures/ssd_meta_arch.py", line 600, in predict
    preprocessed_inputs)
  File "/home/stud/hammadal/custom-model/models/research/object_detection/models/ssd_inception_v2_feature_extractor.py", line 130, in extract_features
    scope=scope)
  File "/home/stud/hammadal/custom-model/models/research/slim/nets/inception_v2.py", line 129, in inception_v2_base
    scope=end_point)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 2784, in separable_convolution2d
    outputs = layer.apply(inputs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1479, in apply
    return self.__call__(inputs, *args, **kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 537, in __call__
    outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 634, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 146, in wrapper
    ), args, kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 446, in converted_call
    return _call_unconverted(f, args, kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 253, in _call_unconverted
    return f(*args, **kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/keras/layers/convolutional.py", line 1658, in call
    data_format=conv_utils.convert_data_format(self.data_format, ndim=4))
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py", line 793, in separable_conv2d
    name=name)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 1953, in conv2d
    name=name)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 1071, in conv2d
    data_format=data_format, dilations=dilations, name=name)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/nfs/student/hammadal/custom-model/tf1.14/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

. Это происходит при запуске его на сервере, где я могу использовать мощность графического процессора. Когда я запускаю сценарий на локальном компьютере, используя только ЦП и размер пакета 1, сценарий выполняется.

Используемый сценарий взят из официального репозитория тензорного потока ЗДЕСЬ . Информация об аппаратном обеспечении сервера выглядит следующим образом:

> OS: Ubuntu x86_64 memory: 503GiB 
> system memory processor: Intel(R)
> Xeon(R) CPU E5-2630 v4 @ 2.20GHz 
> display: GV100GL [Tesla V100 PCIe 32GB]

Библиотеки:

> tensorflow-gpu: 1.14
> numpy: 1.16 
> absl-py 0.9

Я пытался разобраться с прошлыми 2 неделями. Если кто-то может помочь или направить меня, что мне нужно прочитать, я был бы очень рад

1 Ответ

0 голосов
/ 10 мая 2020

Похоже, cuDNN не удалось инициализировать. Что больше связано с TensorFlow. Попробуйте использовать на сервере следующую команду, которая должна правильно установить cuDNN:

conda install tensorflow-gpu
Добро пожаловать на сайт PullRequest, где вы можете задавать вопросы и получать ответы от других членов сообщества.
...