Как удалить униграммы из словаря слов в Python - PullRequest
Новая
       15

Как удалить униграммы из словаря слов в Python

0 голосов
/ 19 января 2019

Я пытаюсь создать облако слов только с биграммами в python. Теперь у меня есть следующий словарь:

Word_dict 

{'delivered later requested_delivered later requested': 0.07590105638848002,
 'delayed delivery_delayed delivery': 0.043280231684707335,
 'guidelines followed_guidelines followed': 0.04056653336980544,
 'delayed pickup_delayed pickup': 0.02733236942769188,
 'delivered later requested_delayed delivery': 0.023815416411579027,
 'delayed delivery_delivered later requested': 0.02332477975624476,
 'guidelines followed_delivered later requested': 0.02131881396186928,
 'delivered later requested_guidelines followed': 0.020793441968104277,
 'delayed pickup_delayed delivery': 0.020619765275950556,
 'delayed delivery_guidelines followed': 0.01998150343228563,
 'delayed delivery_delayed pickup': 0.019464815273128308,
 'guidelines followed_delayed delivery': 0.018900366023628715,
 'delivered later requested_delayed pickup': 0.01870932166225962,
 'delayed pickup_delivered later requested': 0.0185660383912328,
 'guidelines followed_delayed pickup': 0.015148949473108336,
 'delayed pickup_guidelines followed': 0.01475383499845862,
 'super user activity fom_super user activity fom': 0.010490072206084763}

Мне нужно удалить словарей или слова без подчеркивания из словаря. Как это можно сделать.

Ожидаемый результат 

{' requested_delivered ': 0.07590105638848002,
 'delivery_delayed ': 0.043280231684707335,
 'followed_guidelines': 0.04056653336980544,
 'pickup_delayed ': 0.02733236942769188,
 ' requested_delayed ': 0.023815416411579027}

Как это сделать?

Mycode 

def preprocess(x):
    x = re.sub('[^a-z\s]', '', x.lower())                  # get rid of noise
    x = [w for w in x.split() if w not in set(newstopwords)]  # remove stopwords
    return ' '.join(x)

data['Clean_addr'] = data['Reason Code Level 1'].apply(preprocess)

# setup and score the bigrams using the raw frequency.
finder = BigramCollocationFinder.from_words(text_content)
bigram_measures = BigramAssocMeasures()
scored = finder.score_ngrams(bigram_measures.raw_freq)

# By default finder.score_ngrams is sorted, however don't rely on this default behavior.
# Sort highest to lowest based on the score.
scoredList = sorted(scored, key=itemgetter(1), reverse=True)

# word_dict is the dictionary we'll use for the word cloud.
# Load dictionary with the FOR loop below.
# The dictionary will look like this with the bigram and the score from above.
# word_dict = {'bigram A': 0.000697411,
#             'bigram B': 0.000524882}

word_dict = {}

listLen = len(scoredList)

# Get the bigram and make a contiguous string for the dictionary key. 
# Set the key to the scored value. 
for i in range(listLen):
    word_dict['_'.join(scoredList[i][0])] = scoredList[i][1]



# -----

1 Ответ

0 голосов
/ 19 января 2019

Если вы можете гарантировать из своего набора данных, что во всем наборе есть только ноль или одна фраза, содержащая подчеркивание: 

# starting from scoredList in the example above (had to add scored.items()) so that you're iterating over key/value pairs
scoredList = sorted(scored.items(), key=itemgetter(1), reverse=True)

new_data = {}
for key, value in scoredList:
    words = [word for word in key.split(' ') if '_' in word]
    if len(words) == 1:
        new_data[words[0]] = value
    elif len(words) > 1:
        raise ValueError('oh no...')

print(new_data)
...