ИНФОРМАЦИЯ: тензор потока: ошибка сообщена координатору: , 2 root ошибок найдено - PullRequest
1 голос
/ 20 января 2020

Я пытаюсь запустить модель обнаружения объекта с помощью API обнаружения возражений тензорного потока. Моя цель для запуска обнаружения объектов - попытка решить проблему с использованием капчи. Я следую одному уроку для этого. Конфигурация системы: виртуальная машина на Azure GPU - Nivida Tesla K80 RAM - версия тензорного потока 56 - 1.14 Моя модель работает, но она остановилась после 16 итераций, пока эта итерационная модель работает нормально и потери также уменьшаются, но после этого выдают ошибку , Я использую fast_RCNN_resnet_inception_v2_atrous_coco. Я следовал всем путям, которые требуются для выполнения модели. Я даю ввод в пакете 2, больше, чем это дает ошибку исчерпанного ресурса.

INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, 2 root error(s) found.
  (0) Invalid argument: ConcatOp : Dimensions of inputs should match: shape[0] = [1,284,1024,3] vs. shape[1]= [1,296,1024,3]
         [[node concat (defined at /root/workspace/models/research/object_detection/legacy/trainer.py:191) ]]
         [[gradients/FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat/block35_3/Conv2d_1x1/BiasAdd_grad/BiasAddGrad/_6083]]
  (1) Invalid argument: ConcatOp : Dimensions of inputs should match: shape[0] = [1,284,1024,3] vs. shape[1]= [1,296,1024,3]
         [[node concat (defined at /root/workspace/models/research/object_detection/legacy/trainer.py:191) ]]
0 successful operations.
0 derived errors ignored.

Errors may have originated from an input operation.
Input Source operations connected to node concat:
 Preprocessor_1/sub (defined at /root/workspace/models/research/object_detection/models/faster_rcnn_inception_resnet_v2_feature_extractor.py:77)

Input Source operations connected to node concat:
 Preprocessor_1/sub (defined at /root/workspace/models/research/object_detection/models/faster_rcnn_inception_resnet_v2_feature_extractor.py:77)

Original stack trace for 'concat':
  File "legacy/train.py", line 185, in <module>
    tf.app.run()
  File "/opt/sft/miniconda3/envs/ravi_gpu/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/opt/sft/miniconda3/envs/ravi_gpu/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/opt/sft/miniconda3/envs/ravi_gpu/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "/opt/sft/miniconda3/envs/ravi_gpu/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "legacy/train.py", line 181, in main
    graph_hook_fn=graph_rewriter_fn)
  File "/root/workspace/models/research/object_detection/legacy/trainer.py", line 297, in train
    clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
  File "/root/workspace/models/research/slim/deployment/model_deploy.py", line 194, in create_clones
    outputs = model_fn(*args, **kwargs)
  File "/root/workspace/models/research/object_detection/legacy/trainer.py", line 191, in _create_losses
    images = tf.concat(preprocessed_images, 0)
  File "/opt/sft/miniconda3/envs/ravi_gpu/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)
  File "/opt/sft/miniconda3/envs/ravi_gpu/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1299, in concat
    return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
  File "/opt/sft/miniconda3/envs/ravi_gpu/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1256, in concat_v2
    "ConcatV2", values=values, axis=axis, name=name)
  File "/opt/sft/miniconda3/envs/ravi_gpu/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/opt/sft/miniconda3/envs/ravi_gpu/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/opt/sft/miniconda3/envs/ravi_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/opt/sft/miniconda3/envs/ravi_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

I0120 12:23:47.601684 140495582246720 coordinator.py:224] Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, 2 root error(s) found.
  (0) Invalid argument: ConcatOp : Dimensions of inputs should match: shape[0] = [1,284,1024,3] vs. shape[1]= [1,296,1024,3]
         [[node concat (defined at /root/workspace/models/research/object_detection/legacy/trainer.py:191) ]]
         [[gradients/FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat/block35_3/Conv2d_1x1/BiasAdd_grad/BiasAddGrad/_6083]]
  (1) Invalid argument: ConcatOp : Dimensions of inputs should match: shape[0] = [1,284,1024,3] vs. shape[1]= [1,296,1024,3]
         [[node concat (defined at /root/workspace/models/research/object_detection/legacy/trainer.py:191) ]]
0 successful operations.
0 derived errors ignored.
...