У меня есть набор поездов, который я использую, чтобы определить для каждого текста в Твиттере, является ли он положительным или отрицательным (twiterText + twiterTarget) хочу попытаться предсказать, к чему он принадлежит, основываясь на моей модели (twiterTextJ + twiterTargetJ).
twiterText, twiterTarget, twiterTextJ, twiterTargetJ = списки - там добавляется информация в этом сообщении
i получить эту ошибку:
Ошибка IndexEr: индекс 2 выходит за пределы для оси 1 с размером 2
, если я пытаюсь разделить данные поезда на тест и поезд, код работает, но если я принесу новый текст, он падает со стрелкой выше.
код
import re
import json
import keras
import numpy as np
from keras import Sequential
from keras.layers import Dense, Dropout
from keras.preprocessing.text import Tokenizer
import keras.preprocessing.text as kpt
from sklearn.preprocessing import LabelEncoder
twiterTarget = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
twiterText = ['switchfoot http twitpic com 2y1zl awww bummer shoulda got david carr third day', 'upset update facebook texting might cry result school today also blah', 'kenichan dived many times ball managed save 50 rest go bounds', 'whole body feels itchy like fire', 'nationwideclass behaving mad see', 'kwesidei whole crew', 'need hug', 'loltrish hey long time see yes rains bit bit lol fine thanks', 'tatiana_k nope', 'twittera que muera', 'spring break plain city snowing', 'pierced ears', 'caregiving bear watch thought ua loss embarrassing', 'octolinz16 counts idk either never talk anymore', 'smarrison would first gun really though zac snyder doucheclown', 'iamjazzyfizzle wish got watch miss iamlilnicki premiere', 'hollis death scene hurt severely watch film wry directors cut', 'file taxes', 'lettya ahh ive always wanted see rent love soundtrack', 'fakerpattypattz oh dear drinking forgotten table drinks', 'alydesigns day get much done', 'one friend called asked meet mid valley today time sigh', 'angry_barista baked cake ated', 'week going hoped', 'blagh class 8 tomorrow', 'hate call wake people', 'going cry sleep watching marley', 'im sad miss lilly', 'ooooh lol leslie ok leslie get mad', 'meh almost lover exception track gets depressed every time', 'some1 hacked account aim make new one', 'alielayus want go promote gear groove unfornately ride may b going one anaheim may though', 'thought sleeping option tomorrow realizing evaluations morning work afternoon', 'julieebaby awe love 1 miss', 'humpninja cry asian eyes sleep night', 'ok sick spent hour sitting shower cause sick stand held back puke like champ bed', 'cocomix04 ill tell ya story later good day ill workin like three hours', 'missxu sorry bed time came gmt 1 http gd fnge', 'fleurylis either depressing think even want know kids suitcases', 'bed class 8 12 work 12 3 gym 3 5 6 class 6 10 another day gonna fly miss girlfriend', 'really feel like getting today got study tomorrows practical exam', 'reason teardrops guitar one enough break heart', 'sad sad sad know hate feeling wanna sleep still', 'jonathanrknight awww soo wish see finally comfortable im sad missed', 'falling asleep heard tracy girl body found sad heart breaks family', 'viennah yay happy job also means less time', 'checked user timeline blackberry looks like twanking still happening ppl still probs w bgs uids', 'oh man ironing jeancjumbe fave top wear meeting burnt', 'strangely sad lilo samro breaking', 'tea oh sorry think retweeting', 'broadband plan massive broken promise http tinyurl com dcuc33 via www diigo com tautao still waiting broadband', 'localtweeps wow tons replies may unfollow see friends tweets scrolling feed lot', 'duck chicken taking wayyy long hatch', 'put vacation photos online yrs ago pc crashed forget name site', 'need hug', 'andywana sure pos much want dont think trade away company assets sorry andy', 'oanhlove hate happens', 'sad feeling dallas going show gotta say though think shows would use music game mmm', 'ugh 92 degrees tomorrow', 'u move thought u already sd hmmm random u found glad hear yer well', 'batmanyng miss ps3 commission wutcha playing copped blood sand', 'leaving parking lot work', 'life cool', 'sadly though never gotten experience post coitus cigarette never', 'nice day bad rain comes tomorrow 5am', 'starrbby bad around lost job even pay phone bill lmao aw shucks', 'damm back school tomorrow', 'mo jobs money hell min wage 4 f n clams hour', 'katortiz forever see soon', 'lt_algonquin agreed saw failwhale allllll day today', 'jdarter oh haha dude dont really look em unless someone says hey added sorry terrible need pop', 'ninjen sure right need start working nikster jared least', 'really hate people diss bands trace clearly ugly', 'gym attire today puma singlet adidas shorts black business socks leather shoes lucky run cute girls', 'show location http twitpic com 2y2es', 'picnic phone smells like citrus', 'ashleyac donkey sensitive comments nevertheless glad see mug asap charger still awol', 'new csi tonight fml', 'think arms sore tennis', 'wonders someone u like much make unhappy split seccond depressed', 'sleep soon hate saying bye see tomorrow night', 'statravelau got ur newsletter fares really unbelievable shame already booked paid mine', 'missin boo', 'markhardy1974 itm', 'damn chalk chalkboard useless', 'blast getty villa hates sore throat day getting worse', 'msdrama hey missed ya meeting sup mama', 'tummy hurts wonder hypnosis anything working get stop smoking', 'always fat ones', 'januarycrimson sorry babe fam annoys thankfully asleep right muahaha evil laugh', 'hollywoodheat paid attention covered photoshop webpage design class undergrad', 'wednesday b day know 2', 'poor cameron hills', 'pray please ex threatening start sh babies 1st birthday party jerk still headache', 'makeherfamous hmm u really enjoy problems constants u think things find someone ulike', 'strider sick little puppy http apps facebook com dogbook profile view 5248435', 'rylee grace wana go steve party sadly since easter wnt b able 2 much ohh well', 'hey actually one bracket pools bad one money', 'stark follow either work', 'bad nite favorite teams astros spartans lose nite w good', 'body missing northern calif girl found police found remains missing northern california girl http tr im imji', 'mangaaa hope increase capacity fast yesterday pain got fail whale 15 times 2 hours', 'behind classes work', 'watching quot house quot', 'kpreyes remember bum leg strikes back time serious', 'paradisej cool kinds complaints laptop online overheating recalls', 'emily glad mommy done training new job misses http apps facebook com dogbook profile view 6176014', 'would rather first party send bad messages 3rd party send mixed ones sophmore year', 'henkuyinepu overrated', 'marykatherine_q know heard afternoon wondered thing moscow behind times', 'laying bed voice', 'sooo sad killed kutner house whyyyyyyyy', 'jacobsummers sorry tell mea culpa really sorry', 'alliana07 make sense suicide thing refuse believe actually happened', 'salancaster hope ok', 'mercedesashley damn grind inspirational saddening time want stop cuz like u much love', 'hibanick yeah aw know wudnt stand chance', 'ugh cant sleep 1 30am', 'hanging crooners wanna sing sucks', 'erre_sc aaw miss ya im leaving bh tomorrow quot morning quot think aww wanna go beach w u girls', 'pissed asba radio station', 'wednesday b day n know 2', 'know life flipped upside thought head ramen sounds good', 'pain back sides hurt mention crying made fail', 'late night snack glass oj b c quot sickness quot back sleep ugh hate getting sick', 'allyheman big fan camilla belle', 'grum wah see clip must el stupido work filters wait till get puter something else 2 blame ex 4 broke mine', 'week seems get longer longer terms much need much actually going get done', 'cold', 'thecoolestout ehhh weather gonna take turn ugly tomorrow', 'chelserlynn haha cooooold still go show incredible stuff', 'hoping tummy rumbles go away soon', 'knights_ notice told working tomorrow called agency follow said', 'almost bedtime', 'missing babe long alive happy yawwwnn tired love imma try sleep hopefully headstart', 'agh snow']
twiterTargetJ = [4, 0, 0, 2, 2, 0, 4, 2, 0, 0, 0, 2, 0, 0, 4, 0, 4, 0, 4, 0, 0, 4, 4, 2, 2, 2, 2, 2, 0, 0, 2, 2, 2, 2, 0, 2, 2, 2, 2, 0, 0, 0, 0, 0, 0, 2, 0, 2, 2, 2, 2, 2, 2, 2, 4, 4]
twiterTextJ = ['Uncut gems is so fuckin insane', 'Loving this new video highlighting @MOStateSAHE! Proud to be an alum of this stellar program. #cohort8 #sapro \n\nnbPWww9Fbf', "@senrobportman @SenatorCardin @barrons Sure????\n\nTrump's Budget to cut $4.8 TRILLION from Medicare and Medicaid !!… kx2ZBC0IWY", '??', 'OqUG2ALqbm', 'Is it okay if I call you my Prana bae?', '@AaliyahJay Long asf ?? I said the same thing but I want to see it', '??????', '@james_hudgins Tastes too much like cough medicine. I also won’t drink Pepsi products.', 'Starting to get my energy back! Today was my first day out of the house since last Friday’s surgery. Doctor approve… 36HW2bpEK0', '@vbspurs Ounce of prevention... good!!!', '@synthetikminds Are we allowed to bring mini palm trees too??', 'Reason #19 of why I should start bringing my camera along!! I seriously LOVE taking pictures!!\n??\u200d♀️????\u200d♀️?\nDo y’a… 9NsfSsMjmu', '@jamescharles Awe! That’s so sweet!!! As one of these - we thank you for providing us with entertainment - teacha… CZxYZapvq4', '@Andy_Lindahl Ed you sounded like dirty hairy just then', 'Kansas City update: I fell in love with Burnt Ends today. \n\nWhere has Brisket been my whole life? \n\nThanks… P93ArVV0na', '@RodneyCaston @1SnoozyQ "late afternoon"', '@Darbmilne One of my favorite music video moments ever. God, I loved watching this part when it was on MTV. Joe Pe… 7MDtx80DHt', "Now that's freakin scary!", "@PKosena Haha I didn't get it until I saw the OG tweet. ?", '@ancienthistory0 I LOVE THESE SO MF MUCHHHH OMG @abigailraw @DeadbeatsLexx_ @issabopbop', '@Bluto51 @jodaciousjody Consequences absolutely; Censorship? Not hardly...', 'FINAL: @DixieHighTeams pulls away in the second half, beating Green Canyon 61-45 to advance to the 4A semifinals.… npTaXWzqJv', '@aniSashaCohen @AdamMilstein @lsarsour @afbranco @IlhanMN @RashidaTlaib @jeremycorbyn @IfNotNowOrg @BernieSanders… 65jt7bzOCP', '9/10 I prolly won’t miss her.', '@Boz2929 Hell yeah man ❤️', "Looking to score a job at U.S. Army 5th Medical Recruiting Battalion? You're in luck. Click the link in our bio for… uNvUwPI0U8", '@dannymears Not convinced his 47% would all have turned out otherwise tbh.', 'jacobjax excellent closeup magic on the Las Vegas Strip @ Las Vegas On The Strip PTeFTlHS2X', '@fox6now Nice!', 'Hopefully we see the vs Michigan State Purdue team tonight and not the vs Illinois team ? #boilerball #boilerup', 'Oh darn the multiple billion dollar corporation “cant afford” it. Just lost a paying customer.', '@RodneyCaston @MajikaZulJin The Trump Administration will lie to the nation about the Coronavirus!!! Stay tuned fo… ZYuLmTKLBI', '@desireedashaee Ima be in the way Friday ??', '@GBDE2019 true, i didn’t realize how extreme it was, even after the fight he refused to drink water to avoid being… 1fkpHo8o0T', '@nigeriansosa juice', '@Veeplurs Same basically', '@_MichelleSteph ?', 'Sell the VIX\nSell the VIX.....', 'Dysfunctional when taking pics together since ‘01 and still goin’ strong ?\u200d♀️ @ Breckenridge, Colorado VGvgqfAnZH', 'Axe throwing Best Deal in Colorado. $35 bucks gets you 2 beers and 60 mins per person at @twentybrew. Choice from o… HrfgxNYw4t', 'A little bit more #challenegecoin #love for our #specialforceschallengecoin #primuscoin #instavideo… KEDsWUr3sq', 'I was closing my trunk and my hand slipped and I broke a nail, my REAL nail ???', 'lemme see sumn. OKAY OKAY?', 'I love play action on first down ❤️', 'Arsenal make beans', '? TODAYS SALES! ?\nRARE COLLECTIBLES!\n#EBAY STORE LINK IN BIO ⬆️ #TwinLakesSilverFrontier #silver #gold #coins… TayxgvMBJK', '@advanunez @1017_moneymitch @Jj_mendez3 Africans and American negroes are completely different nations of people. The more u know', '⛅️ #worksmarternotharder \n.\n.\n.\n.\n.\n? @briankphotog \n.\n.\n.\n.\n.\n.\n#photooftheday #thursday #tb #tbt #throwback… UzhCaXrrBj', 'kDdcZZtvaN', 'vqdOTjeetW', '@finebaum Marcus Dupree', 'Have you ever had a role like "Laboratory Assistant - Rotating 3rd shift"? What did you like most about that job?… un13nJQSWu', 'Let me bleed instead of you.', '‘Sigh’: EPA Officials Were Flabbergasted By Trump’s Toilet Rant AQEoyd6gZC via @TPM Remember Flush 1… FJ3ds59W2t', 'Still mad I’ve never met @KREAYSHAWN irl']
def deeplearningModel2():
X_train = twiterText
X_test = twiterTextJ
y_train = twiterTarget
y_test = twiterTargetJ
tokenizer = Tokenizer(num_words=2500)
tokenizer.fit_on_texts(twiterText)
tokenizer.fit_on_texts(twiterTextJ)
dictionary = tokenizer.word_index
wordsList = []
wordsListTest = []
for text in X_train:
wordIndices = [dictionary[word] for word in kpt.text_to_word_sequence(text)]
wordsList.append(wordIndices)
for text in X_test:
wordIndices = [dictionary[word] for word in kpt.text_to_word_sequence(text)]
wordsListTest.append(wordIndices)
wordsList = np.asarray(wordsList)
wordsListTest = np.asarray(wordsListTest)
XTrainVec = tokenizer.sequences_to_matrix(wordsList, mode='binary')
XTestVec = tokenizer.sequences_to_matrix(wordsListTest, mode='binary')
lb_make = LabelEncoder()
YTrainNums2 = lb_make.fit_transform(y_train)
YTestNums2 = lb_make.fit_transform(y_test)
YTrainVec = keras.utils.to_categorical(YTrainNums2, 2)
YTestVec = keras.utils.to_categorical(YTestNums2, 2)
model = Sequential()
model.add(Dense(512, input_shape=(2500,), activation='sigmoid', name='dense_1'))
model.add(Dropout(0.5))
model.add(Dense(256, activation='sigmoid', name='dense_2'))
model.add(Dropout(0.5))
model.add(Dense(2, activation='softmax', name='dense_3'))
model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy'])
print(model.summary())
model.fit(XTrainVec, YTrainVec, batch_size=128, epochs=5, verbose=1,validation_data=(XTestVec,YTestVec),shuffle=True)
scores = model.evaluate(XTestVec, YTestVec)
print("Deep Learning Model Accuracy: %.2f%%" % (scores[1]*100))
deeplearningModel2()