Как я могу сделать токенизацию предложений

Question

Это код, который я использовал для sent_tokenize

import nltk
from nltk.tokenize import sent_tokenize
sent_tokenize(comments1)

Набор данных

И я использовал массив, чтобы получитьпредложения по одному, но это не сработало

Arr=sent_tokenize(comments1)
Arr
Arr[0]

И когда я использую Arr[1], появляется эта ошибка

IndexError                                
Traceback (most recent call last) <ipython-input-27-c15dd30f2746> in <module>
----> 1 Arr[1]

IndexError: list index out of range

Answer 1

Токенайзер NLTK по умолчанию не распознает предложения здесь, потому что отсутствует последняя пунктуация. Вы можете добавить его перед каждой новой строкой "\n".

Например:

comments1 = comments1.replace("\n", ".\n")
tokens = sent_tokenize(comments1)
for token in tokens:
    print("sentence: " + token)

Вы получите что-то подобное (усеченное для удобства чтения):

sentence: 1, Opposition MP Namal Rajapaksa questions Environment Ministe [...] Sirisena over Wilpattu deforestation issue.
sentence: 2, but he should remember that it all started with his dad and [...] a coma in that days .
sentence: 3, Opposition MP Namal Rajapaksa questions Environment Ministe [...] Sirisena over Wilpattu deforestation issue.
sentence: 4, Pawu meya ba ba meyage  [...] 
sentence: 5, We visited Wilpaththu in August 2013 These are some of the  [...] deforestation of Wilpattu as Srilankans .
sentence: 6, Mamath wiruddai .
sentence: 7, Yeah we should get together and do something.
sentence: B, Mama Kamathyi oka kawada hari wenna tiyana deyak .
sentence: 9, Yes Channa Karunaratne we should stand agaist this act dnt  [...] as per news spreading Pls .
sentence: 10, LTTE eken elawala daapu oya minissunta awurudu 30kin passe [...] sathungena balaneka thama manussa kama .
sentence: 11, Shanthar mahaththayo ow aththa eminisunta idam denna one g [...] vikalpa yojana gena evi kiyala.
sentence: 12, You are wrong They must be given lands No one opposes it W [...]

Answer 2

Прочитайте комментарии в следующем.

# Standard sentence tokenizer.
def sent_tokenize(text, language='english'):
    """
    Return a sentence-tokenized copy of *text*,
    using NLTK's recommended sentence tokenizer
    (currently :class:`.PunktSentenceTokenizer`
    for the specified language).

    :param text: text to split into sentences
    :param language: the model name in the Punkt corpus
    """
    tokenizer = load('tokenizers/punkt/{0}.pickle'.format(language))
    return tokenizer.tokenize(text)


def tokenize(self, text, realign_boundaries=True):
    """
    Given a text, returns a list of the sentences in that text.
    """
    return list(self.sentences_from_text(text, realign_boundaries))

Поскольку language='english' принимает !, ?, . ... в качестве конца предложения, он работает для добавления comments1 = comments1.replace('\n', '. ') до sent_tokenize(comments1).

Возможно, ваш случай дублируется как токенизатор предложений nltk, новые строки следует рассматривать как границу предложения

Answer 3

NLTK sent_tokenize работает с хорошо отформатированным текстом. Я думаю, что вы ищете регулярное выражение:

import re

comments_str = "1,Opposition MP Namal Rajapaksa questions Environment Minister President Sirisena over Wilpattu deforestation issue\nbut he should remember that it all started with his dad and uncle and might be he was in a coma in that days \n3, Opposition on MP Namal Rajapaksa questions Environment Minister President Sirisena over Wilpattu deforestation issue\n4,Pawu meya ba meyage thathata oka deddi kiyana thibbane"
comments = re.split(r'(?:^\d+,)|(?:\n\d+,)', comments_str)
print(comments)

Выходы:

[
    '',
    'Opposition MP Namal Rajapaksa questions Environment Minister President Sirisena over Wilpattu deforestation issue\nbut he should remember that it all started with his dad and uncle and might be he was in a coma in that days ',
    ' Opposition on MP Namal Rajapaksa questions Environment Minister President Sirisena over Wilpattu deforestation issue',
    'Pawu meya ba meyage thathata oka deddi kiyana thibbane'

]