Я тренирую модель, используя Mobilenet-SSD-v2, она тренируется некоторое время, затем пытается оценить и зависает.
Я запускаю tenorflow-gpu 1.14, внутри образа докера tensorflow/tensorflow:latest-gpu,Я использую RTX 2060 на Ubuntu 19.04. Я использую последний API обнаружения объектов из этого репозитория git: https://github.com/tensorflow/models.
Я пытался установить throttle_secs в model_lib.py, он ничего не делал. Я все еще могу тренироваться, но каждый раз, когда он пытается оценить, мне нужно перезапускать контейнер докера.
Я использую только код, предоставленный репозиторием git. Я начинаю тренировку, используя команду ниже.
PIPELINE_CONFIG_PATH=/tensorflow/models/research/face/pipeline.config
MODEL_DIR=/tensorflow/models/research/face/training/
NUM_TRAIN_STEPS=50000
SAMPLE_1_OF_N_EVAL_EXAMPLES=1
python object_detection/model_main.py \
--pipeline_config_path=${PIPELINE_CONFIG_PATH} \
--model_dir=${MODEL_DIR} \
--num_train_steps=${NUM_TRAIN_STEPS} \
--sample_1_of_n_eval_examples=$SAMPLE_1_OF_N_EVAL_EXAMPLES \
--alsologtostderr
Я ожидал, что она продолжит тренировку. Но вместо этого я просто застреваю и мне нужно перезагрузить.
I1002 18:28:30.106040 139663203059520 evaluation.py:255] Starting evaluation at 2019-10-02T18:28:30Z
I1002 18:28:30.717183 139663203059520 monitored_session.py:240] Graph was finalized.
2019-10-02 18:28:30.717937: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-02 18:28:30.718182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.71
pciBusID: 0000:0a:00.0
2019-10-02 18:28:30.718232: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-10-02 18:28:30.718251: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-10-02 18:28:30.718263: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-10-02 18:28:30.718279: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-10-02 18:28:30.718295: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-10-02 18:28:30.718309: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-10-02 18:28:30.718326: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-10-02 18:28:30.718401: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-02 18:28:30.718655: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-02 18:28:30.718861: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-10-02 18:28:30.718888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-02 18:28:30.718898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2019-10-02 18:28:30.718907: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2019-10-02 18:28:30.718992: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-02 18:28:30.719242: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-02 18:28:30.719460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4946 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:0a:00.0, compute capability: 7.5)
I1002 18:28:30.720419 139663203059520 saver.py:1280] Restoring parameters from /tensorflow/models/research/face/training/model.ckpt-10756
I1002 18:28:32.285661 139663203059520 session_manager.py:500] Running local_init_op.
I1002 18:28:32.408489 139663203059520 session_manager.py:502] Done running local_init_op.