Сбой TensorFlow для задачи NER с использованием больших данных

Question

Я использую реализацию NER в NER .с этой конфигурацией:

# embeddings
dim_word = 300
dim_char = 100

# training
train_embeddings = False
nepochs          = 15
dropout          = 0.5
batch_size       = 20
lr_method        = "adam"
lr               = 0.001
lr_decay         = 0.9
clip             = -1 # if negative, no clipping
nepoch_no_imprv  = 3

# model hyperparameters
hidden_size_char = 100 # lstm on chars
hidden_size_lstm = 300 # lstm on word embeddings

Размер обучающих данных, которые у меня есть, составляет 3000 предложений.Когда я просто использую ок.2500 предложений обучение сделано без каких-либо проблем.Когда я использую все предложения обучения, я получаю эту ошибку:

 Epoch 1 out of 15
 58/115 [==============>...............] - ETA: 734s - train loss: 15.6397Traceback (most recent call last):
  File "**/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "**/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "**/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[1,21,0] = 136 is not in [0, 136)
         [[{{node chars/char_embeddings}} = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@train_step/Adam/update_chars/_char_embeddings/AssignSub"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](chars/_char_embeddings/read, _arg_char_ids_0_0, chars/char_embeddings/axis)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 26, in <module>
    main()
  File "train.py", line 23, in main
    model.train(train, dev)
  File "**/base_model.py", line 121, in train
    score = self.run_epoch(train, dev, epoch)
  File "**/model/ner_model.py", line 287, in run_epoch
    [self.train_op, self.loss, self.merged], feed_dict=fd)
  File "**/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "**/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/***/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "**/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[1,21,0] = 136 is not in [0, 136)
         [[node chars/char_embeddings (defined at **/model/ner_model.py:128)  = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@train_step/Adam/update_chars/_char_embeddings/AssignSub"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](chars/_char_embeddings/read, _arg_char_ids_0_0, chars/char_embeddings/axis)]]

Информация об оборудовании:

x86_64 x86_64 x86_64 GNU/Linux
CPU MHz:               1200.671
CPU max MHz:           3500.0000
Memory                  503 GB