Я пытаюсь использовать инструмент Medical Concept Annotation.Документацию можно найти здесь: https://github.com/w-is-h/MedCAT
Код, который у меня есть:
from medcat.cat import CAT
from medcat.utils.vocab import Vocab
from medcat.prepare_cdb import PrepareCDB
from medcat.cdb import CDB
vocab = Vocab()
#Load the vocab model
vocab.load_dict('C:/Users/as/Desktop/Data/med_ann_norm_dict.dat')
#build a special CDB from a .csv file
preparator = PrepareCDB(vocab=vocab)
csv_paths = ['C:/Users/as/Desktop/Data/ICD10Codes.csv']
cdb = preparator.prepare_csvs(csv_paths)
#Annotate the text
doc = text
cat = CAT(cdb=cdb, vocab=vocab)
#Entities are in
doc_spacy._.ents
Трассировка:
File "<ipython-input-26-32d167ab52db>", line 1, in <module>
runfile('C:/Users/as/Desktop/Data/Preprocessing.py', wdir='C:/Users/as/Desktop/Data')
File "C:\Users\as\AppData\Local\Continuum\anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 786, in runfile
execfile(filename, namespace)
File "C:\Users\as\AppData\Local\Continuum\anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/as/Desktop/Data/Preprocessing.py", line 198, in <module>
main()
File "C:/Users/as/Desktop/Data/Preprocessing.py", line 193, in main
preproc.parse()
File "C:/Users/as/Desktop/Data/Preprocessing.py", line 135, in parse
self.medcat(text)
File "C:/Users/as/Desktop/Data/Preprocessing.py", line 181, in medcat
cdb = preparator.prepare_csvs(csv_paths)
File "C:\Users\as\AppData\Local\Continuum\anaconda3\lib\site-packages\medcat\prepare_cdb.py", line 53, in prepare_csvs
df = pandas.read_csv(csv_path, sep=sep)
File "C:\Users\as\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\parsers.py", line 702, in parser_f
return _read(filepath_or_buffer, kwds)
File "C:\Users\as\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\parsers.py", line 435, in _read
data = parser.read(nrows)
File "C:\Users\as\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\parsers.py", line 1139, in read
ret = self._engine.read(nrows)
File "C:\Users\as\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\parsers.py", line 1995, in read
data = self._reader.read(nrows)
File "pandas/_libs/parsers.pyx", line 899, in pandas._libs.parsers.TextReader.read
File "pandas/_libs/parsers.pyx", line 914, in pandas._libs.parsers.TextReader._read_low_memory
File "pandas/_libs/parsers.pyx", line 991, in pandas._libs.parsers.TextReader._read_rows
File "pandas/_libs/parsers.pyx", line 1123, in pandas._libs.parsers.TextReader._convert_column_data
File "pandas/_libs/parsers.pyx", line 1176, in pandas._libs.parsers.TextReader._convert_tokens
File "pandas/_libs/parsers.pyx", line 1299, in pandas._libs.parsers.TextReader._convert_with_dtype
File "pandas/_libs/parsers.pyx", line 1315, in pandas._libs.parsers.TextReader._string_convert
File "pandas/_libs/parsers.pyx", line 1553, in pandas._libs.parsers._string_box_utf8
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 3: invalid start byte
CSV-файл, которым я являюсьпытается загрузить имеет 2 столбца.Первый содержит коды ICD10, а второй содержит термин, поясняющий код.
Пример 2 строк файла CSV:
ICD10 ICD10 Term
A00 Cholera
A000 Cholera due to Vibrio cholerae 01, biovar cholerae