Произошла ошибка при завершении итератора GeneratorDataset: Отменено: Операция отменена - Tensorflow

Question

Я действительно новичок в TensorFlow, построении моделей и обучении. Однако я следовал учебному пособию, и все шло хорошо, пока в один прекрасный момент я не получил следующую ошибку:

2020-04-29 17: 24: 35.235550: W tensflowflow / core / kernels / data / generator_dataset_op. cc: 103] Ошибка при завершении итератора GeneratorDataset: Отменено: Операция была отменена

Понятия не имею, что является причиной ошибки. Я использую следующий код:

import tensorflow as tf
from tensorflow.keras.optimizers import RMSprop
import keras_preprocessing
from keras_preprocessing import image
from keras.preprocessing.image import ImageDataGenerator
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import os
from PIL import Image

training_datagen = ImageDataGenerator(rescale=1. / 255)
validation_datagen = ImageDataGenerator(rescale=1. / 255)

# Here I am giving the path of the images to train the model
train_dir = r"C:\Users\User\Desktop\Project\Project\dataset\train"
train_gen = training_datagen.flow_from_directory(train_dir, target_size=(150, 150),     class_mode="categorical")

val_dir = r"C:\Users\User\Desktop\Project\Project\dataset\validation"
val_gen = training_datagen.flow_from_directory(val_dir, target_size=(150, 150), class_mode="categorical")

# Here I am training the model with individual fruits
train_apple_dir = r"C:\Users\User\Desktop\Project\Project\dataset\train\Apple"
train_banana_dir = r"C:\Users\User\Desktop\Project\Project\dataset\train\Banana"

# printing the number of apples in train dataset
number_apples_train = len(os.listdir(train_apple_dir))
print("total training apple images:", number_apples_train)

number_banana_train = len(os.listdir(train_banana_dir))
print("total training apple images:", number_banana_train)

# Here I am getting the first 10 names of apple images
apple_names = os.listdir(train_apple_dir)
print(apple_names[:10])

# Building the model
model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(64, (3, 3), activation="relu", input_shape=(150, 150, 3)),
    tf.keras.layers.MaxPooling2D(2, 2),
    tf.keras.layers.Conv2D(64, (3, 3), activation="relu"),
    tf.keras.layers.MaxPooling2D(2, 2),
    tf.keras.layers.Conv2D(128, (3, 3), activation="relu"),
    tf.keras.layers.MaxPooling2D(2, 2),
    tf.keras.layers.Conv2D(128, (3, 3), activation="relu"),
    tf.keras.layers.MaxPooling2D(2, 2),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(512, activation="relu"),
    tf.keras.layers.Dense(15, activation="softmax")
])
model.summary()

model.compile(loss="categorical_crossentropy", optimizer='rmsprop', metrics=['accuracy'])

fruit_model = model.fit(train_gen, epochs=1, verbose=1, validation_data=val_gen, workers=10)

Полная трассировка ошибок:

C:\Users\User\anaconda3\envs\project-env\python.exe             
C:/Users/User/Desktop/Project/2ndYearProject/fruit_classifier.py
Using TensorFlow backend.
Found 7765 images belonging to 15 classes.
Found 7765 images belonging to 15 classes.
total training apple images: 492
total training apple images: 490
['0_100.jpg', '100_100.jpg', '101_100.jpg', '102_100.jpg', '103_100.jpg',     '104_100.jpg', '105_100.jpg', '106_100.jpg', '107_100.jpg', '108_100.jpg']
2020-04-29 17:21:17.562203: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions     that this TensorFlow binary was not compiled to use: AVX AVX2
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 148, 148, 64)      1792      
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 74, 74, 64)        0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 72, 72, 64)        36928     
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 36, 36, 64)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 34, 34, 128)       73856     
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 17, 17, 128)       0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 15, 15, 128)       147584    
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 7, 7, 128)         0         
_________________________________________________________________
flatten (Flatten)            (None, 6272)              0         
_________________________________________________________________
dense (Dense)                (None, 512)               3211776   
_________________________________________________________________
dense_1 (Dense)              (None, 15)                7695      
=================================================================
Total params: 3,479,631
Trainable params: 3,479,631
Non-trainable params: 0
_________________________________________________________________
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 243 steps, validate for 243 steps

  1/243 [..............................] - ETA: 6:49 - loss: 2.7038 - accuracy: 0.1875
  2/243 [..............................] - ETA: 4:34 - loss: 3.4685 - accuracy: 0.1406
  3/243 [..............................] - ETA: 3:47 - loss: 3.1995 - accuracy: 0.1562
  4/243 [..............................] - ETA: 3:26 - loss: 3.0967 - accuracy: 0.1172
  5/243 [..............................] - ETA: 3:11 - loss: 3.0149 - accuracy: 0.1187
  6/243 [..............................] - ETA: 3:02 - loss: 2.9531 - accuracy: 0.1094
  7/243 [..............................] - ETA: 2:54 - loss: 2.9210 - accuracy: 0.0982
  8/243 [..............................] - ETA: 2:49 - loss: 2.8854 - accuracy: 0.1094
  9/243 [>.............................] - ETA: 2:46 - loss: 2.8430 - accuracy: 0.1181
 10/243 [>.............................] - ETA: 2:46 - loss: 2.8042 - accuracy: 0.1312
 11/243 [>.............................] - ETA: 2:43 - loss: 2.8623 - accuracy: 0.1307
 12/243 [>.............................] - ETA: 2:41 - loss: 2.8341 - accuracy: 0.1354
 13/243 [>.............................] - ETA: 2:39 - loss: 2.7907 - accuracy: 0.1611
 14/243 [>.............................] - ETA: 2:37 - loss: 2.7307 - accuracy: 0.1763
 15/243 [>.............................] - ETA: 2:36 - loss: 2.6982 - accuracy: 0.1917
 16/243 [>.............................] - ETA: 2:35 - loss: 2.6755 - accuracy: 0.1855
 17/243 [=>............................] - ETA: 2:34 - loss: 2.6318 - accuracy: 0.1930
 18/243 [=>............................] - ETA: 2:33 - loss: 2.5757 - accuracy: 0.2083
 19/243 [=>............................] - ETA: 2:31 - loss: 2.5932 - accuracy: 0.2122
 20/243 [=>............................] - ETA: 2:30 - loss: 2.5570 - accuracy: 0.2234
 21/243 [=>............................] - ETA: 2:29 - loss: 2.5350 - accuracy: 0.2321
 22/243 [=>............................] - ETA: 2:28 - loss: 2.4881 - accuracy: 0.2500
 23/243 [=>............................] - ETA: 2:26 - loss: 2.4220 - accuracy: 0.2704
 24/243 [=>............................] - ETA: 2:25 - loss: 2.3835 - accuracy: 0.2799
 25/243 [==>...........................] - ETA: 2:24 - loss: 2.3830 - accuracy: 0.2825
 26/243 [==>...........................] - ETA: 2:22 - loss: 2.3481 - accuracy: 0.2885
 27/243 [==>...........................] - ETA: 2:21 - loss: 2.2924 - accuracy: 0.3090
 28/243 [==>...........................] - ETA: 2:20 - loss: 2.2524 - accuracy: 0.3270
 29/243 [==>...........................] - ETA: 2:18 - loss: 2.2276 - accuracy: 0.3287
 30/243 [==>...........................] - ETA: 2:17 - loss: 2.2073 - accuracy: 0.3344
 31/243 [==>...........................] - ETA: 2:16 - loss: 2.1791 - accuracy: 0.3407
 32/243 [==>...........................] - ETA: 2:15 - loss: 2.1395 - accuracy: 0.3516
 33/243 [===>..........................] - ETA: 2:14 - loss: 2.0994 - accuracy: 0.3636
 34/243 [===>..........................] - ETA: 2:13 - loss: 2.0554 - accuracy: 0.3759
 35/243 [===>..........................] - ETA: 2:12 - loss: 2.0124 - accuracy: 0.3893
 36/243 [===>..........................] - ETA: 2:11 - loss: 1.9763 - accuracy: 0.4002
 37/243 [===>..........................] - ETA: 2:10 - loss: 1.9455 - accuracy: 0.4079
 38/243 [===>..........................] - ETA: 2:09 - loss: 1.9066 - accuracy: 0.4194
 39/243 [===>..........................] - ETA: 2:08 - loss: 1.8678 - accuracy: 0.4311
 40/243 [===>..........................] - ETA: 2:07 - loss: 1.8342 - accuracy: 0.4406
 41/243 [====>.........................] - ETA: 2:07 - loss: 1.8034 - accuracy: 0.4505
 42/243 [====>.........................] - ETA: 2:06 - loss: 1.7719 - accuracy: 0.4591
 43/243 [====>.........................] - ETA: 2:05 - loss: 1.7409 - accuracy: 0.4680
 44/243 [====>.........................] - ETA: 2:04 - loss: 1.7135 - accuracy: 0.4766
 45/243 [====>.........................] - ETA: 2:03 - loss: 1.6833 - accuracy: 0.4829
 46/243 [====>.........................] - ETA: 2:02 - loss: 1.6609 - accuracy: 0.4901
 47/243 [====>.........................] - ETA: 2:01 - loss: 1.6461 - accuracy: 0.4930
 48/243 [====>.........................] - ETA: 2:00 - loss: 1.6200 - accuracy: 0.5003
 49/243 [=====>........................] - ETA: 1:59 - loss: 1.6031 - accuracy: 0.5048
 50/243 [=====>........................] - ETA: 1:59 - loss: 1.5824 - accuracy: 0.5110
 51/243 [=====>........................] - ETA: 1:58 - loss: 1.5569 - accuracy: 0.5188
 52/243 [=====>........................] - ETA: 1:57 - loss: 1.5354 - accuracy: 0.5239
 53/243 [=====>........................] - ETA: 1:56 - loss: 1.5183 - accuracy: 0.5294
 54/243 [=====>........................] - ETA: 1:56 - loss: 1.5067 - accuracy: 0.5323
 55/243 [=====>........................] - ETA: 1:55 - loss: 1.4871 - accuracy: 0.5363
 56/243 [=====>........................] - ETA: 1:54 - loss: 1.4643 - accuracy: 0.5441
 57/243 [======>.......................] - ETA: 1:53 - loss: 1.4433 - accuracy: 0.5494
 58/243 [======>.......................] - ETA: 1:53 - loss: 1.4205 - accuracy: 0.5566
 59/243 [======>.......................] - ETA: 1:52 - loss: 1.4007 - accuracy: 0.5626
 60/243 [======>.......................] - ETA: 1:51 - loss: 1.3805 - accuracy: 0.5689
 61/243 [======>.......................] - ETA: 1:50 - loss: 1.3604 - accuracy: 0.5750
 62/243 [======>.......................] - ETA: 1:50 - loss: 1.3424 - accuracy: 0.5793
 63/243 [======>.......................] - ETA: 1:49 - loss: 1.3232 - accuracy: 0.5855
 64/243 [======>.......................] - ETA: 1:48 - loss: 1.3069 - accuracy: 0.5896
 65/243 [=======>......................] - ETA: 1:48 - loss: 1.2911 - accuracy: 0.5945
 66/243 [=======>......................] - ETA: 1:47 - loss: 1.2776 - accuracy: 0.5983
 67/243 [=======>......................] - ETA: 1:46 - loss: 1.2648 - accuracy: 0.6029
 68/243 [=======>......................] - ETA: 1:45 - loss: 1.2506 - accuracy: 0.6065
 69/243 [=======>......................] - ETA: 1:45 - loss: 1.2385 - accuracy: 0.6099
 70/243 [=======>......................] - ETA: 1:44 - loss: 1.2282 - accuracy: 0.6137
 71/243 [=======>......................] - ETA: 1:44 - loss: 1.2153 - accuracy: 0.6174
 72/243 [=======>......................] - ETA: 1:43 - loss: 1.1989 - accuracy: 0.6228
 73/243 [========>.....................] - ETA: 1:42 - loss: 1.1864 - accuracy: 0.6262
 74/243 [========>.....................] - ETA: 1:42 - loss: 1.1716 - accuracy: 0.6313
 75/243 [========>.....................] - ETA: 1:41 - loss: 1.1584 - accuracy: 0.6346
 76/243 [========>.....................] - ETA: 1:40 - loss: 1.1454 - accuracy: 0.6382
 77/243 [========>.....................] - ETA: 1:40 - loss: 1.1318 - accuracy: 0.6425
 78/243 [========>.....................] - ETA: 1:39 - loss: 1.1181 - accuracy: 0.6471
 79/243 [========>.....................] - ETA: 1:38 - loss: 1.1044 - accuracy: 0.6516
 80/243 [========>.....................] - ETA: 1:38 - loss: 1.0913 - accuracy: 0.6559
 81/243 [=========>....................] - ETA: 1:37 - loss: 1.0794 - accuracy: 0.6598
 82/243 [=========>....................] - ETA: 1:36 - loss: 1.0698 - accuracy: 0.6636
 83/243 [=========>....................] - ETA: 1:36 - loss: 1.0581 - accuracy: 0.6673
 84/243 [=========>....................] - ETA: 1:35 - loss: 1.0458 - accuracy: 0.6713
 85/243 [=========>....................] - ETA: 1:34 - loss: 1.0352 - accuracy: 0.6744
 86/243 [=========>....................] - ETA: 1:34 - loss: 1.0258 - accuracy: 0.6775
 87/243 [=========>....................] - ETA: 1:33 - loss: 1.0338 - accuracy: 0.6780
 88/243 [=========>....................] - ETA: 1:32 - loss: 1.0462 - accuracy: 0.6756
 89/243 [=========>....................] - ETA: 1:32 - loss: 1.0428 - accuracy: 0.6757
 90/243 [==========>...................] - ETA: 1:31 - loss: 1.0373 - accuracy: 0.6772
 91/243 [==========>...................] - ETA: 1:30 - loss: 1.0289 - accuracy: 0.6801
 92/243 [==========>...................] - ETA: 1:30 - loss: 1.0195 - accuracy: 0.6829
 93/243 [==========>...................] - ETA: 1:29 - loss: 1.0091 - accuracy: 0.6860
 94/243 [==========>...................] - ETA: 1:28 - loss: 0.9992 - accuracy: 0.6894
 95/243 [==========>...................] - ETA: 1:28 - loss: 0.9892 - accuracy: 0.6926
 96/243 [==========>...................] - ETA: 1:27 - loss: 0.9792 - accuracy: 0.6959
 97/243 [==========>...................] - ETA: 1:26 - loss: 0.9696 - accuracy: 0.6990
 98/243 [===========>..................] - ETA: 1:26 - loss: 0.9599 - accuracy: 0.7021
 99/243 [===========>..................] - ETA: 1:25 - loss: 0.9503 - accuracy: 0.7051
100/243 [===========>..................] - ETA: 1:25 - loss: 0.9410 - accuracy: 0.7081
101/243 [===========>..................] - ETA: 1:24 - loss: 0.9319 - accuracy: 0.7110
102/243 [===========>..................] - ETA: 1:23 - loss: 0.9229 - accuracy: 0.7138
103/243 [===========>..................] - ETA: 1:23 - loss: 0.9140 - accuracy: 0.7166
104/243 [===========>..................] - ETA: 1:22 - loss: 0.9052 - accuracy: 0.7193
105/243 [===========>..................] - ETA: 1:21 - loss: 0.8969 - accuracy: 0.7217
106/243 [============>.................] - ETA: 1:21 - loss: 0.8900 - accuracy: 0.7238
107/243 [============>.................] - ETA: 1:20 - loss: 0.8842 - accuracy: 0.7260
108/243 [============>.................] - ETA: 1:20 - loss: 0.8769 - accuracy: 0.7283
109/243 [============>.................] - ETA: 1:19 - loss: 0.8694 - accuracy: 0.7305
110/243 [============>.................] - ETA: 1:18 - loss: 0.8619 - accuracy: 0.7327
111/243 [============>.................] - ETA: 1:18 - loss: 0.8544 - accuracy: 0.7351
112/243 [============>.................] - ETA: 1:17 - loss: 0.8468 - accuracy: 0.7375
113/243 [============>.................] - ETA: 1:17 - loss: 0.8393 - accuracy: 0.7398
114/243 [=============>................] - ETA: 1:16 - loss: 0.8320 - accuracy: 0.7421
115/243 [=============>................] - ETA: 1:15 - loss: 0.8249 - accuracy: 0.7443
116/243 [=============>................] - ETA: 1:15 - loss: 0.8178 - accuracy: 0.7466
117/243 [=============>................] - ETA: 1:14 - loss: 0.8109 - accuracy: 0.7487
118/243 [=============>................] - ETA: 1:14 - loss: 0.8041 - accuracy: 0.7509
119/243 [=============>................] - ETA: 1:13 - loss: 0.7975 - accuracy: 0.7530
120/243 [=============>................] - ETA: 1:12 - loss: 0.7940 - accuracy: 0.7545
121/243 [=============>................] - ETA: 1:12 - loss: 0.7913 - accuracy: 0.7558
122/243 [==============>...............] - ETA: 1:11 - loss: 0.7934 - accuracy: 0.7562
123/243 [==============>...............] - ETA: 1:11 - loss: 0.8107 - accuracy: 0.7529
124/243 [==============>...............] - ETA: 1:10 - loss: 0.8141 - accuracy: 0.7528
125/243 [==============>...............] - ETA: 1:09 - loss: 0.8127 - accuracy: 0.7531
126/243 [==============>...............] - ETA: 1:09 - loss: 0.8088 - accuracy: 0.7545
127/243 [==============>...............] - ETA: 1:08 - loss: 0.8031 - accuracy: 0.7565
128/243 [==============>...............] - ETA: 1:08 - loss: 0.7984 - accuracy: 0.7576
129/243 [==============>...............] - ETA: 1:07 - loss: 0.7944 - accuracy: 0.7588
130/243 [===============>..............] - ETA: 1:06 - loss: 0.7885 - accuracy: 0.7607
131/243 [===============>..............] - ETA: 1:06 - loss: 0.7832 - accuracy: 0.7623
132/243 [===============>..............] - ETA: 1:05 - loss: 0.7775 - accuracy: 0.7641
133/243 [===============>..............] - ETA: 1:04 - loss: 0.7724 - accuracy: 0.7656
134/243 [===============>..............] - ETA: 1:04 - loss: 0.7668 - accuracy: 0.7674
135/243 [===============>..............] - ETA: 1:03 - loss: 0.7613 - accuracy: 0.7691
136/243 [===============>..............] - ETA: 1:03 - loss: 0.7557 - accuracy: 0.7708
137/243 [===============>..............] - ETA: 1:02 - loss: 0.7503 - accuracy: 0.7725
138/243 [================>.............] - ETA: 1:01 - loss: 0.7450 - accuracy: 0.7741
139/243 [================>.............] - ETA: 1:01 - loss: 0.7402 - accuracy: 0.7755
140/243 [================>.............] - ETA: 1:00 - loss: 0.7355 - accuracy: 0.7771
141/243 [================>.............] - ETA: 1:00 - loss: 0.7307 - accuracy: 0.7785
142/243 [================>.............] - ETA: 59s - loss: 0.7256 - accuracy: 0.7801 
143/243 [================>.............] - ETA: 58s - loss: 0.7206 - accuracy: 0.7816
144/243 [================>.............] - ETA: 58s - loss: 0.7156 - accuracy: 0.7831
145/243 [================>.............] - ETA: 57s - loss: 0.7107 - accuracy: 0.7846
146/243 [=================>............] - ETA: 57s - loss: 0.7058 - accuracy: 0.7861
147/243 [=================>............] - ETA: 56s - loss: 0.7011 - accuracy: 0.7876
148/243 [=================>............] - ETA: 55s - loss: 0.6963 - accuracy: 0.7890
149/243 [=================>............] - ETA: 55s - loss: 0.6917 - accuracy: 0.7904
150/243 [=================>............] - ETA: 54s - loss: 0.6870 - accuracy: 0.7918
151/243 [=================>............] - ETA: 54s - loss: 0.6825 - accuracy: 0.7932
152/243 [=================>............] - ETA: 53s - loss: 0.6780 - accuracy: 0.7946
153/243 [=================>............] - ETA: 52s - loss: 0.6736 - accuracy: 0.7959
154/243 [==================>...........] - ETA: 52s - loss: 0.6696 - accuracy: 0.7970
155/243 [==================>...........] - ETA: 51s - loss: 0.6655 - accuracy: 0.7983
156/243 [==================>...........] - ETA: 51s - loss: 0.6613 - accuracy: 0.7996
157/243 [==================>...........] - ETA: 50s - loss: 0.6571 - accuracy: 0.8009
158/243 [==================>...........] - ETA: 49s - loss: 0.6530 - accuracy: 0.8022
159/243 [==================>...........] - ETA: 49s - loss: 0.6489 - accuracy: 0.8034
160/243 [==================>...........] - ETA: 48s - loss: 0.6450 - accuracy: 0.8047
161/243 [==================>...........] - ETA: 48s - loss: 0.6413 - accuracy: 0.8057
162/243 [===================>..........] - ETA: 47s - loss: 0.6543 - accuracy: 0.8049
163/243 [===================>..........] - ETA: 46s - loss: 0.6824 - accuracy: 0.8035
164/243 [===================>..........] - ETA: 46s - loss: 0.6813 - accuracy: 0.8029
165/243 [===================>..........] - ETA: 45s - loss: 0.6780 - accuracy: 0.8038
166/243 [===================>..........] - ETA: 45s - loss: 0.6743 - accuracy: 0.8049
167/243 [===================>..........] - ETA: 44s - loss: 0.6705 - accuracy: 0.8061
168/243 [===================>..........] - ETA: 43s - loss: 0.6666 - accuracy: 0.8073
169/243 [===================>..........] - ETA: 43s - loss: 0.6627 - accuracy: 0.8084
170/243 [===================>..........] - ETA: 42s - loss: 0.6589 - accuracy: 0.8095
171/243 [====================>.........] - ETA: 42s - loss: 0.6552 - accuracy: 0.8107
172/243 [====================>.........] - ETA: 41s - loss: 0.6515 - accuracy: 0.8118
173/243 [====================>.........] - ETA: 40s - loss: 0.6478 - accuracy: 0.8129
174/243 [====================>.........] - ETA: 40s - loss: 0.6441 - accuracy: 0.8139
175/243 [====================>.........] - ETA: 39s - loss: 0.6405 - accuracy: 0.8150
176/243 [====================>.........] - ETA: 39s - loss: 0.6370 - accuracy: 0.8160
177/243 [====================>.........] - ETA: 38s - loss: 0.6334 - accuracy: 0.8171
178/243 [====================>.........] - ETA: 37s - loss: 0.6304 - accuracy: 0.8178
179/243 [=====================>........] - ETA: 37s - loss: 0.6281 - accuracy: 0.8186
180/243 [=====================>........] - ETA: 36s - loss: 0.6248 - accuracy: 0.8196
181/243 [=====================>........] - ETA: 36s - loss: 0.6215 - accuracy: 0.8206
182/243 [=====================>........] - ETA: 35s - loss: 0.6181 - accuracy: 0.8216
183/243 [=====================>........] - ETA: 34s - loss: 0.6148 - accuracy: 0.8226
184/243 [=====================>........] - ETA: 34s - loss: 0.6114 - accuracy: 0.8235
185/243 [=====================>........] - ETA: 33s - loss: 0.6082 - accuracy: 0.8245
186/243 [=====================>........] - ETA: 33s - loss: 0.6051 - accuracy: 0.8255
187/243 [======================>.......] - ETA: 32s - loss: 0.6018 - accuracy: 0.8264
188/243 [======================>.......] - ETA: 32s - loss: 0.5988 - accuracy: 0.8271
189/243 [======================>.......] - ETA: 31s - loss: 0.5962 - accuracy: 0.8279
190/243 [======================>.......] - ETA: 30s - loss: 0.5946 - accuracy: 0.8281
191/243 [======================>.......] - ETA: 30s - loss: 0.5949 - accuracy: 0.8277
192/243 [======================>.......] - ETA: 29s - loss: 0.6007 - accuracy: 0.8273
193/243 [======================>.......] - ETA: 29s - loss: 0.5988 - accuracy: 0.8279
194/243 [======================>.......] - ETA: 28s - loss: 0.5958 - accuracy: 0.8288
195/243 [=======================>......] - ETA: 27s - loss: 0.5931 - accuracy: 0.8297
196/243 [=======================>......] - ETA: 27s - loss: 0.5901 - accuracy: 0.8305
197/243 [=======================>......] - ETA: 26s - loss: 0.5875 - accuracy: 0.8314
198/243 [=======================>......] - ETA: 26s - loss: 0.5847 - accuracy: 0.8323
199/243 [=======================>......] - ETA: 25s - loss: 0.5818 - accuracy: 0.8331
200/243 [=======================>......] - ETA: 24s - loss: 0.5789 - accuracy: 0.8339
201/243 [=======================>......] - ETA: 24s - loss: 0.5760 - accuracy: 0.8348
202/243 [=======================>......] - ETA: 23s - loss: 0.5733 - accuracy: 0.8356
203/243 [========================>.....] - ETA: 23s - loss: 0.5705 - accuracy: 0.8364
204/243 [========================>.....] - ETA: 22s - loss: 0.5677 - accuracy: 0.8372
205/243 [========================>.....] - ETA: 22s - loss: 0.5650 - accuracy: 0.8380
206/243 [========================>.....] - ETA: 21s - loss: 0.5622 - accuracy: 0.8388
207/243 [========================>.....] - ETA: 20s - loss: 0.5595 - accuracy: 0.8396
208/243 [========================>.....] - ETA: 20s - loss: 0.5568 - accuracy: 0.8403
209/243 [========================>.....] - ETA: 19s - loss: 0.5542 - accuracy: 0.8411
210/243 [========================>.....] - ETA: 19s - loss: 0.5516 - accuracy: 0.8419
211/243 [=========================>....] - ETA: 18s - loss: 0.5490 - accuracy: 0.8426
212/243 [=========================>....] - ETA: 18s - loss: 0.5464 - accuracy: 0.8433
213/243 [=========================>....] - ETA: 17s - loss: 0.5438 - accuracy: 0.8441
214/243 [=========================>....] - ETA: 16s - loss: 0.5413 - accuracy: 0.8448
215/243 [=========================>....] - ETA: 16s - loss: 0.5388 - accuracy: 0.8455
216/243 [=========================>....] - ETA: 15s - loss: 0.5363 - accuracy: 0.8463
217/243 [=========================>....] - ETA: 15s - loss: 0.5338 - accuracy: 0.8470
218/243 [=========================>....] - ETA: 14s - loss: 0.5314 - accuracy: 0.8477
219/243 [==========================>...] - ETA: 13s - loss: 0.5289 - accuracy: 0.8484
220/243 [==========================>...] - ETA: 13s - loss: 0.5265 - accuracy: 0.8491
221/243 [==========================>...] - ETA: 12s - loss: 0.5242 - accuracy: 0.8497
222/243 [==========================>...] - ETA: 12s - loss: 0.5218 - accuracy: 0.8504
223/243 [==========================>...] - ETA: 11s - loss: 0.5195 - accuracy: 0.8511
224/243 [==========================>...] - ETA: 11s - loss: 0.5172 - accuracy: 0.8518
225/243 [==========================>...] - ETA: 10s - loss: 0.5149 - accuracy: 0.8524
226/243 [==========================>...] - ETA: 9s - loss: 0.5126 - accuracy: 0.8531 
227/243 [===========================>..] - ETA: 9s - loss: 0.5103 - accuracy: 0.8537
228/243 [===========================>..] - ETA: 8s - loss: 0.5081 - accuracy: 0.8544
229/243 [===========================>..] - ETA: 8s - loss: 0.5059 - accuracy: 0.8550
230/243 [===========================>..] - ETA: 7s - loss: 0.5039 - accuracy: 0.8555
231/243 [===========================>..] - ETA: 6s - loss: 0.5034 - accuracy: 0.8560
232/243 [===========================>..] - ETA: 6s - loss: 0.5079 - accuracy: 0.8554
233/243 [===========================>..] - ETA: 5s - loss: 0.5107 - accuracy: 0.8556
234/243 [===========================>..] - ETA: 5s - loss: 0.5091 - accuracy: 0.8560
235/243 [============================>.] - ETA: 4s - loss: 0.5071 - accuracy: 0.8566
236/243 [============================>.] - ETA: 4s - loss: 0.5050 - accuracy: 0.8572
237/243 [============================>.] - ETA: 3s - loss: 0.5029 - accuracy: 0.8578
238/243 [============================>.] - ETA: 2s - loss: 0.5009 - accuracy: 0.8584
239/243 [============================>.] - ETA: 2s - loss: 0.4988 - accuracy: 0.8590
240/243 [============================>.] - ETA: 1s - loss: 0.4968 - accuracy: 0.8596
241/243 [============================>.] - ETA: 1s - loss: 0.4947 - accuracy: 0.8601
242/243 [============================>.] - ETA: 0s - loss: 0.4927 - accuracy: 0.8607
243/243 [==============================] - 197s 811ms/step - loss: 0.4907 -     accuracy: 0.8613 - val_loss: 0.0068 - val_accuracy: 0.9994
2020-04-29 17:24:35.235550: W     tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled

Process finished with exit code 0

Answer 1

Кажется, это известная проблема с текущими сообщениями о возникновении, даже в самых последних версиях TensorFlow. По-видимому, это связано с параллелизмом и / или стратегией распределения, используемой для генератора данных. Один простой обходной путь - использовать только одного работника, то есть workers=1 (который является значением по умолчанию, если не установлено), при вызове model.fit.