Компьютер:
- ASUS TUF X570-PLUS
- Ryzen 3900x
- 32 ГБ ОЗУ 3200 МГц
- RTX 2080 Super (драйвер 445,87)
- Монитор, подключенный через DisplayPort к графическому процессору
Программное обеспечение:
- Windows 10 64-бит
- CUDA 10.1
- CuDNN 7.6.5
- Python 3.7.7
- tensorflow-gpu 2.1
Во время При обучении сверточной сети я заметил, что скорость обучения снижается, когда другие процессы используют графический процессор. Даже открытие и отображение изображения в несколько мегапикселей или воспроизведение видео на YouTube приводит к замедлению обучения, даже если загрузка графического процессора не приближается к 100%. Хуже того, это дросселирование продолжается бесконечно. Мне нужно перезапустить обучение, чтобы исправить это.
Кажется, не имеет значения, что это за процесс, даже перемещение / изменение размера windows вокруг дисплея может вызвать это.
Существуют ли настройки windows / nvidia / tensorflow / keras, которые предотвращают такое поведение?
Ниже приведен пример: первые 12 эпох были успешными, но затем я открыл изображение, а затем скорость обучения примерно вдвое.
2020-05-06 12:10:10.515701: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-05-06 12:10:10.519444: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:09:00.0 name: GeForce RTX 2080 SUPER computeCapability: 7.5
coreClock: 1.83GHz coreCount: 48 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 462.00GiB/s
2020-05-06 12:10:10.524140: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-05-06 12:10:10.529424: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-05-06 12:10:10.531637: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-05-06 12:10:10.534027: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-05-06 12:10:10.536298: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-05-06 12:10:10.541765: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-05-06 12:10:10.544161: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-05-06 12:10:10.546728: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-05-06 12:10:10.975411: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-06 12:10:10.977534: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2020-05-06 12:10:10.979233: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2020-05-06 12:10:10.981434: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6265 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 SUPER, pci bus id: 0000:09:00.0, compute capability: 7.5)
Epoch 1/200
2020-05-06 12:10:20.462588: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-05-06 12:10:21.381864: W tensorflow/stream_executor/gpu/redzone_allocator.cc:312] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation. This message will be only logged once.
2020-05-06 12:10:21.420974: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
499/500 [============================>.] - ETA: 0s - loss: 0.1194 - categorical_accuracy: 0.9704 - AUC: 0.9809
Epoch 1/200
500/500 [==============================] - 128s 255ms/step - loss: 0.1194 - categorical_accuracy: 0.9704 - AUC: 0.9809 - val_loss: 0.0980 - val_categorical_accuracy: 0.9807 - val_AUC: 0.9798
Epoch 2/200
500/500 [==============================] - 114s 227ms/step - loss: 0.0863 - categorical_accuracy: 0.9785 - AUC: 0.9912 - val_loss: 0.1194 - val_categorical_accuracy: 0.9713 - val_AUC: 0.9849
Epoch 3/200
500/500 [==============================] - 114s 227ms/step - loss: 0.0715 - categorical_accuracy: 0.9792 - AUC: 0.9949 - val_loss: 0.1062 - val_categorical_accuracy: 0.9786 - val_AUC: 0.9852
Epoch 4/200
500/500 [==============================] - 114s 227ms/step - loss: 0.0665 - categorical_accuracy: 0.9798 - AUC: 0.9957 - val_loss: 0.1063 - val_categorical_accuracy: 0.9756 - val_AUC: 0.9875
Epoch 5/200
500/500 [==============================] - 114s 227ms/step - loss: 0.0638 - categorical_accuracy: 0.9804 - AUC: 0.9960 - val_loss: 0.1046 - val_categorical_accuracy: 0.9784 - val_AUC: 0.9871
Epoch 6/200
500/500 [==============================] - 114s 227ms/step - loss: 0.0618 - categorical_accuracy: 0.9806 - AUC: 0.9963 - val_loss: 0.0963 - val_categorical_accuracy: 0.9786 - val_AUC: 0.9892
Epoch 7/200
500/500 [==============================] - 113s 226ms/step - loss: 0.0601 - categorical_accuracy: 0.9811 - AUC: 0.9966 - val_loss: 0.1096 - val_categorical_accuracy: 0.9792 - val_AUC: 0.9864
Epoch 8/200
500/500 [==============================] - 113s 226ms/step - loss: 0.0588 - categorical_accuracy: 0.9813 - AUC: 0.9967 - val_loss: 0.1054 - val_categorical_accuracy: 0.9783 - val_AUC: 0.9873
Epoch 9/200
500/500 [==============================] - 113s 226ms/step - loss: 0.0569 - categorical_accuracy: 0.9818 - AUC: 0.9969 - val_loss: 0.1091 - val_categorical_accuracy: 0.9791 - val_AUC: 0.9869
Epoch 10/200
500/500 [==============================] - 113s 225ms/step - loss: 0.0556 - categorical_accuracy: 0.9824 - AUC: 0.9970 - val_loss: 0.1202 - val_categorical_accuracy: 0.9790 - val_AUC: 0.9853
Epoch 11/200
500/500 [==============================] - 113s 227ms/step - loss: 0.0551 - categorical_accuracy: 0.9824 - AUC: 0.9971 - val_loss: 0.1077 - val_categorical_accuracy: 0.9789 - val_AUC: 0.9875
Epoch 12/200
500/500 [==============================] - 112s 224ms/step - loss: 0.0533 - categorical_accuracy: 0.9829 - AUC: 0.9973 - val_loss: 0.0836 - val_categorical_accuracy: 0.9780 - val_AUC: 0.9926
Epoch 13/200
500/500 [==============================] - 113s 225ms/step - loss: 0.0543 - categorical_accuracy: 0.9828 - AUC: 0.9971 - val_loss: 0.0867 - val_categorical_accuracy: 0.9788 - val_AUC: 0.9916
Epoch 14/200
500/500 [==============================] - 282s 563ms/step - loss: 0.0538 - categorical_accuracy: 0.9827 - AUC: 0.9972 - val_loss: 0.0935 - val_categorical_accuracy: 0.9785 - val_AUC: 0.9903
Epoch 15/200
500/500 [==============================] - 300s 601ms/step - loss: 0.0538 - categorical_accuracy: 0.9827 - AUC: 0.9973 - val_loss: 0.0905 - val_categorical_accuracy: 0.9789 - val_AUC: 0.9908
Использование графического процессора увеличивается с ~ 30% до ~ 20% после замедления. Память, выделенная тензорным потоком, остается прежней.