У меня есть набор данных в формате CSV, который выглядит следующим образом:
1,dont like the natives
2,Keep it local always
2,Karibu kenya
Метка 1 обозначает разжигание ненависти, а 2 обозначает положительный.
Вотмой код:
import numpy as np
import csv
import tensorflow as tf
from tensorflow.keras.layers import (
Masking, LSTM, Dense, TimeDistributed, Activation)
def tokenize(text):
"""
Change text string into number and
make sure they resulting np.array is of the same size
"""
Tokenizer = tf.keras.preprocessing.text.Tokenizer
t = Tokenizer()
t.fit_on_texts(text)
tokenized_text = t.texts_to_sequences(text)
tokenized_text = [item for sublist in tokenized_text for item in sublist]
return np.resize(np.array(tokenized_text), (1, 30))
x_train = []
y_train = []
# Reading data from CSV
with open('data.csv') as csv_file:
csv_reader = csv.reader(csv_file, delimiter=',')
line_count = 0
for row in csv_reader:
line_count = line_count+1
if line_count == 1:
continue
# Tokenize input data
tokenized = tokenize(row[1])
x_train.append(tokenized)
y_train.append(row[0])
x_train = np.array(x_train).astype('float32')
y_train = np.array(y_train).astype('float32')
x_test = x_train[:3]
y_test = y_train[:3]
input_shape = x_train[0].shape
output_shape = y_train.shape
batch_size = len(y_train)
model = tf.keras.models.Sequential()
model.add(Masking(mask_value=-1, input_shape=input_shape))
model.add(LSTM(batch_size, dropout=0.2))
model.add(Dense(input_dim=batch_size, units=output_shape[-1]))
model.add(Activation('softmax'))
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_train, y_train, epochs=100, batch_size=batch_size)
model.evaluate(x_test, y_test)
for text in ["Karibu kenya", ]:
tokenized_text = tokenize(text)
prediction = model.predict(tokenized_text, batch_size=1, verbose=1)
# Results
print("Text: {}: Prediction: {}".format(text, prediction))
Остальная часть кода работает нормально, но я не могу запустить model.predict(tokenized_text, batch_size=1, verboze=1)
Вместо этого я получаю следующую ошибку:
Epoch 97/100
19/19 [==============================] - 0s 196us/sample - loss: 0.8753 - accuracy: 0.5789
Epoch 98/100
19/19 [==============================] - 0s 246us/sample - loss: 0.8525 - accuracy: 0.6842
Epoch 99/100
19/19 [==============================] - 0s 169us/sample - loss: 0.7961 - accuracy: 0.6842
Epoch 100/100
19/19 [==============================] - 0s 191us/sample - loss: 0.7745 - accuracy: 0.7368
3/3 [==============================] - 0s 115ms/sample - loss: 0.5518 - accuracy: 1.0000
Traceback (most recent call last):
File "start.py", line 65, in <module>
prediction = model.predict(tokenized_text, batch_size=1, verbose=1)
File "/home/felix/Projects/keras/.env/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 821, in predict
use_multiprocessing=use_multiprocessing)
File "/home/felix/Projects/keras/.env/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 705, in predict
x, check_steps=True, steps_name='steps', steps=steps)
File "/home/felix/Projects/keras/.env/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 2428, in _standardize_user_data
exception_prefix='input')
File "/home/felix/Projects/keras/.env/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_utils.py", line 512, in standardize_input_data
'with shape ' + str(data_shape))
ValueError: Error when checking input: expected masking_input to have 3 dimensions, but got array with shape (1, 30)
Не уверен, что я делаю не так.Я пытался изменить форму данных, но все еще не работает.
Заранее спасибо.