У меня есть схема обучения CNN, в которой я определил методы для импорта и предварительной обработки данных с использованием tf.data.Dataset API и, в конечном итоге, обучения с помощью keras fit.
Моя модель обучается на больших массивах по 5 тусклых размеров, где каждый массив хранится в файле .mat и использует импортированные и предварительно обработанные ниже файлы. Я определяю набор данных для обучения, проверки и тестирования в классе, который я использую только для инициализации самого набора данных. структура набора данных:
output_types = dict()
for key, val in self.list_IDs[0].items():
output_types[key] = tf.convert_to_tensor(val).dtype
gen = data_gen(self.list_IDs, self.shuffle) ##class that has generator as method
ds = tf.data.Dataset.from_generator(generator=gen.file_and_config_gen, output_types=output_types)
ds = ds.map(load_data, num_parallel_calls=self.num_parallel_calls)
ds = ds.batch(self.batch_size, drop_remainder=False)
ds = ds.repeat(self.epochs)
в приведенном выше примере все атрибуты относятся к классу, в котором определен набор данных. генератор определил в классе следующим образом:
class data_gen:
def __init__(self, file_list, shuffle):
self.file_list = file_list
self.shuffle = shuffle
def file_and_config_gen(self):
print('starting generator and shuffle list')
self.indexes = np.arange(len(self.file_list))
if self.shuffle:
np.random.shuffle(self.indexes)
for i in self.indexes:
ID = self.file_list[int(i)]
yield ID
список с именем file_list и list_IDs содержит список словарей, в которых каждый словарь имеет настройки импорта и предварительную обработку для определенного c файла.
функция карты, определенная ниже:
def load_data(self, config_dict):
x, y = tf.py_function(_read_mat, [config_dict['file_path'], self.x_field, self.y_field], (tf.complex128, tf.float64))
x = tf.reshape(x, self.initial_shape_import)
y = tf.reshape(y, self.y_shape_import)
if self.aug_en:
begin = [0] * len(x.shape.as_list())
begin[self.x_aug_ch] = config_dict['x_aug_slice'][0]
size = x.shape.as_list()
size_aug_dim = self.dim_x[self.x_aug_ch]
size[self.x_aug_ch] = size_aug_dim
x = tf.slice(x, begin, size)
begin = [0] * len(y.shape.as_list())
begin[self.y_aug_ch] = config_dict['y_aug_slice'][0]
size = y.shape.as_list()
size_aug_dim = self.dim_y[self.y_aug_ch]
size[self.y_aug_ch] = size_aug_dim
y = tf.slice(y, begin, size)
if self.make_y_on_x_grid:
y = tf.image.resize(y, [self.lateral_grid, self.lateral_grid], method=tf.image.ResizeMethod.NEAREST_NEIGHBOR,
preserve_aspect_ratio=True)
y.set_shape(self.dim_y)
x.set_shape(self.dim_x)
if self.x_type == 'abs':
x = tf.math.abs(x)
if self.standardize_norm_data:
x = (x - self.mean_data[0]) / tf.sqrt(self.std_data[0])
return x, y
else:
real_x, imag_x = tf.math.real(x), tf.math.imag(x)
if self.standardize_norm_data:
real_x = (real_x - self.mean_data[0]) / tf.sqrt(self.std_data[0])
imag_x = (imag_x - self.mean_data[1]) / tf.sqrt(self.std_data[1])
return (real_x, imag_x), y
где на самом деле чтение из файла .mat:
from scipy.io import loadmat
def _read_mat(filepath, x_field_name, y_field_name):
f = loadmat(filepath.numpy().decode('ASCII'), variable_names=[x_field_name.numpy().decode('ASCII'), y_field_name.numpy().decode('ASCII')])
return f[x_field_name.numpy().decode('ASCII')], f[y_field_name.numpy().decode('ASCII')]
Я использую загрузку в парралеле с помощью функции карты, так как я потерялся с чередовать. Я тренируюсь вызывать keras.fit:
model.fit(x=x, y=y, validation_data=validation_data, epochs=epochs, steps_per_epoch=steps_per_epoch, workers=workers,
validation_steps=validation_steps, validation_split=validation_split, batch_size=batch_size)
, где x, validation_data - это указанный выше набор данных, но с использованием разных list_ID для каждого. (в конечном итоге каждый объект в списке представляет собой отдельный образец).
Я запускаю описанное выше и, начиная с последнего пакета (я определил шаги меньшего размера, чтобы исключить последний неполный пакет), я получаю ошибки, связанные с исчерпанием память.
основные проблемы: 1. Память полностью загружается, и программа останавливается.
2. программа также заходит в тупик, когда обучение останавливается, когда у нее достаточно памяти
3. генератор вызывается с самого начала несколько раз, когда повторяются и шаги
генератор соответствует эпохе и количеству пакетов.
ниже - это печать для строка cmd, с которой начинается обучение. Я добавил печать объектов набора данных и размеров обучающих тестов. Я использую tf 2.1, python 3.7 на windows 10 и unix. Я использую nvidia geforce gtx 1080 Ti, а на другом компьютере я использую станцию DGX с Tesla V100.
Train for 74 steps, validate for 7 steps
Epoch 1/10
starting generator and shuffle list
2020-05-06 23:02:10.306681: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-05-06 23:02:10.617387: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-05-06 23:02:11.837448: W tensorflow/stream_executor/gpu/redzone_allocator.cc:312] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation. This message will be only logged once.
73/74 [============================>.] - ETA: 6s - loss: 0.0068 - accuracy: 0.8548 - mse: 0.0068 starting generator and shuffle list
starting generator and shuffle list
2020-05-06 23:11:39.180994: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 17179869184 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:11:39.187845: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 17179869184
2020-05-06 23:11:39.319512: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 15461881856 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:11:39.324774: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 15461881856
2020-05-06 23:11:39.440502: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 13915693056 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:11:39.444384: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 13915693056
74/74 [==============================] - 595s 8s/step - loss: 0.0068 - accuracy: 0.8550 - mse: 0.0068 - val_loss: 0.0046 - val_accuracy: 0.8561 - val_mse: 0.0046
Epoch 2/10
6/74 [=>............................] - ETA: 1:59 - loss: 0.0056 - accuracy: 0.8549 - mse: 0.0056starting generator and shuffle list
16/74 [=====>........................] - ETA: 6:59 - loss: 0.0056 - accuracy: 0.8533 - mse: 0.0056starting generator and shuffle list
20/74 [=======>......................] - ETA: 7:32 - loss: 0.0054 - accuracy: 0.8540 - mse: 0.00542020-05-06 23:14:57.484569: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:14:57.488560: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:14:57.806151: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:14:57.810867: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:14:58.171997: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:14:58.175640: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:14:58.556121: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:14:58.561653: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:14:58.934121: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:14:58.938117: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:14:59.221201: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:14:59.226916: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:14:59.525388: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:14:59.530471: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
21/74 [=======>......................] - ETA: 7:42 - loss: 0.0054 - accuracy: 0.8540 - mse: 0.00542020-05-06 23:15:08.166446: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:08.170612: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:15:08.473786: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:08.477095: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:15:08.723715: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:08.727838: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:15:08.978101: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:08.985016: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:15:09.224867: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:09.229381: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
22/74 [=======>......................] - ETA: 7:38 - loss: 0.0053 - accuracy: 0.8541 - mse: 0.00532020-05-06 23:15:16.810263: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:16.814654: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:15:17.068734: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:17.075101: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:15:27.351469: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:27.359544: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:15:27.639494: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:27.643745: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:15:27.646560: W tensorflow/core/common_runtime/bfc_allocator.cc:424] Allocator (gpu_host_bfc) ran out of memory trying to allocate 889.89MiB (rounded to 933120000). Current allocation summary follows.
2020-05-06 23:15:27.653039: I tensorflow/core/common_runtime/bfc_allocator.cc:894] BFCAllocator dump for gpu_host_bfc
2020-05-06 23:15:27.655249: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (256): Total Chunks: 18, Chunks in use: 18. 4.5KiB allocated for chunks. 4.5KiB in use in bin. 188B client-requested in use in bin.
2020-05-06 23:15:27.659356: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (512): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.663243: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1024): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.667674: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.671611: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.675500: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.679394: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.685740: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.689666: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.694015: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.699658: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.704774: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (524288): Total Chunks: 1, Chunks in use: 0. 1019.5KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.709846: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.718097: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.723683: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.729806: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8388608): Total Chunks: 16, Chunks in use: 15. 180.32MiB allocated for chunks. 169.34MiB in use in bin. 164.79MiB client-requested in use in bin.
2020-05-06 23:15:27.735560: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.739513: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.743982: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.752531: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (134217728): Total Chunks: 1, Chunks in use: 0. 182.97MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.758239: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (268435456): Total Chunks: 33, Chunks in use: 30. 27.31GiB allocated for chunks. 26.16GiB in use in bin. 26.07GiB client-requested in use in bin.
2020-05-06 23:15:27.764473: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Bin for 889.89MiB was 256.00MiB, Chunk State:
2020-05-06 23:15:27.768356: I tensorflow/core/common_runtime/bfc_allocator.cc:923] Size: 268.21MiB | Requested Size: 16B | in_use: 0 | bin_num: 20, prev: Size: 889.89MiB | Requested Size: 889.89MiB | in_use: 1 | bin_num: -1
2020-05-06 23:15:27.772774: I tensorflow/core/common_runtime/bfc_allocator.cc:923] Size: 375.33MiB | Requested Size: 4B | in_use: 0 | bin_num: 20, prev: Size: 889.89MiB | Requested Size: 889.89MiB | in_use: 1 | bin_num: -1
2020-05-06 23:15:27.781140: I tensorflow/core/common_runtime/bfc_allocator.cc:923] Size: 536.43MiB | Requested Size: 16B | in_use: 0 | bin_num: 20, prev: Size: 889.89MiB | Requested Size: 889.89MiB | in_use: 1 | bin_num: -1
2020-05-06 23:15:27.785531: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 1048576
2020-05-06 23:15:27.787679: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400000 of size 256 next 1
2020-05-06 23:15:27.790016: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400100 of size 256 next 2
2020-05-06 23:15:27.793309: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400200 of size 256 next 3
2020-05-06 23:15:27.796157: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400300 of size 256 next 4
2020-05-06 23:15:27.799000: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400400 of size 256 next 5
2020-05-06 23:15:27.802126: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400500 of size 256 next 6
2020-05-06 23:15:27.804496: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400600 of size 256 next 7
2020-05-06 23:15:27.810142: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400700 of size 256 next 8
2020-05-06 23:15:27.812679: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400800 of size 256 next 9
2020-05-06 23:15:27.815004: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400900 of size 256 next 10
2020-05-06 23:15:27.817357: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400a00 of size 256 next 11
2020-05-06 23:15:27.819784: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400b00 of size 256 next 12
2020-05-06 23:15:27.822918: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400c00 of size 256 next 13
2020-05-06 23:15:27.826392: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400d00 of size 256 next 14
2020-05-06 23:15:27.829365: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400e00 of size 256 next 26
2020-05-06 23:15:27.831838: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400f00 of size 256 next 24
2020-05-06 23:15:27.834410: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203401000 of size 256 next 23
2020-05-06 23:15:27.836764: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203401100 of size 256 next 18
2020-05-06 23:15:27.842107: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 203401200 of size 1043968 next 18446744073709551615
2020-05-06 23:15:27.844875: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 1073741824
2020-05-06 23:15:27.847025: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203600000 of size 1025280000 next 44
2020-05-06 23:15:27.849456: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 2407c8800 of size 11520000 next 47
2020-05-06 23:15:27.852011: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 2412c5000 of size 11520000 next 51
2020-05-06 23:15:27.855183: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 241dc1800 of size 11520000 next 54
2020-05-06 23:15:27.857892: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 2428be000 of size 13901824 next 18446744073709551615
2020-05-06 23:15:27.860911: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 1073741824
2020-05-06 23:15:27.863559: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 243600000 of size 933120000 next 43
2020-05-06 23:15:27.866005: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 27afe4800 of size 11520000 next 33
2020-05-06 23:15:27.868392: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 27bae1000 of size 11520000 next 35
2020-05-06 23:15:27.874496: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 27c5dd800 of size 11520000 next 39
2020-05-06 23:15:27.877340: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 27d0da000 of size 11520000 next 32
2020-05-06 23:15:27.879780: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 27dbd6800 of size 11520000 next 25
2020-05-06 23:15:27.882212: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 27e6d3000 of size 11520000 next 55
2020-05-06 23:15:27.885197: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 27f1cf800 of size 11520000 next 56
2020-05-06 23:15:27.887578: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 27fccc000 of size 11520000 next 63
2020-05-06 23:15:27.889964: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 2807c8800 of size 11520000 next 64
2020-05-06 23:15:27.892345: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 2812c5000 of size 11520000 next 67
2020-05-06 23:15:27.894728: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 281dc1800 of size 11520000 next 70
2020-05-06 23:15:27.897160: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 2828be000 of size 13901824 next 18446744073709551615
2020-05-06 23:15:27.901880: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 2147483648
2020-05-06 23:15:27.907589: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 283600000 of size 933120000 next 52
2020-05-06 23:15:27.910125: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 2bafe4800 of size 933120000 next 22
2020-05-06 23:15:27.912860: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 2f29c9000 of size 281243648 next 18446744073709551615
2020-05-06 23:15:27.915652: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 4294967296
2020-05-06 23:15:27.918425: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 303a00000 of size 933120000 next 48
2020-05-06 23:15:27.920888: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 33b3e4800 of size 933120000 next 40
2020-05-06 23:15:27.923344: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 372dc9000 of size 933120000 next 30
2020-05-06 23:15:27.925741: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 3aa7ad800 of size 933120000 next 36
2020-05-06 23:15:27.928094: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 3e2192000 of size 562487296 next 18446744073709551615
2020-05-06 23:15:27.930778: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 8589934592
2020-05-06 23:15:27.936683: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 403a00000 of size 933120000 next 41
2020-05-06 23:15:27.939106: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 43b3e4800 of size 933120000 next 45
2020-05-06 23:15:27.941529: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 472dc9000 of size 933120000 next 34
2020-05-06 23:15:27.943952: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 4aa7ad800 of size 933120000 next 38
2020-05-06 23:15:27.946331: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 4e2192000 of size 933120000 next 42
2020-05-06 23:15:27.949448: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 519b76800 of size 933120000 next 31
2020-05-06 23:15:27.952226: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 55155b000 of size 933120000 next 19
2020-05-06 23:15:27.954579: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 588f3f800 of size 933120000 next 21
2020-05-06 23:15:27.956960: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 5c0924000 of size 933120000 next 27
2020-05-06 23:15:27.959313: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 5f8308800 of size 191854592 next 18446744073709551615
2020-05-06 23:15:27.961996: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 12524123136
2020-05-06 23:15:27.967098: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 603a00000 of size 933120000 next 50
2020-05-06 23:15:27.969450: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 63b3e4800 of size 933120000 next 53
2020-05-06 23:15:27.971800: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 672dc9000 of size 933120000 next 46
2020-05-06 23:15:27.974361: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 6aa7ad800 of size 933120000 next 58
2020-05-06 23:15:27.976726: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 6e2192000 of size 933120000 next 16
2020-05-06 23:15:27.980288: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 719b76800 of size 933120000 next 57
2020-05-06 23:15:27.983390: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 75155b000 of size 933120000 next 29
2020-05-06 23:15:27.986713: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 788f3f800 of size 933120000 next 60
2020-05-06 23:15:27.989122: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7c0924000 of size 933120000 next 62
2020-05-06 23:15:27.991528: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7f8308800 of size 933120000 next 65
2020-05-06 23:15:27.995357: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 82fced000 of size 933120000 next 61
2020-05-06 23:15:28.000221: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 8676d1800 of size 933120000 next 68
2020-05-06 23:15:28.003281: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 89f0b6000 of size 933120000 next 66
2020-05-06 23:15:28.005785: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 8d6a9a800 of size 393563136 next 18446744073709551615
2020-05-06 23:15:28.008652: I tensorflow/core/common_runtime/bfc_allocator.cc:955] Summary of in-use Chunks by size:
2020-05-06 23:15:28.011532: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 18 Chunks of size 256 totalling 4.5KiB
2020-05-06 23:15:28.013833: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 13 Chunks of size
здесь эта строка повторяется, и программа вылетает