"Нет достаточного значения для распаковки" при загрузке набора данных - Allennlp _read - PullRequest
Новая
       51

"Нет достаточного значения для распаковки" при загрузке набора данных - Allennlp _read

0 голосов
/ 01 ноября 2019

Я пытаюсь использовать библиотеку Allennlp для выполнения NER. Библиотека прекрасно работает с conll2003 и другими базами данных, которые имеют только сущности и токены (мне пришлось обновить функцию _read для того же). Но функция возвращает «ValueError: недостаточно значений для распаковки (ожидается 2, получено 1)», если я пытаюсь использовать свой собственный набор данных. Я сравнил форматирование, специальные символы, пробелы и даже имена файлов, но не смог найти никаких проблем. Это образец из набора данных, который работал, 

O   show
O   me
O   films
O   with
B-ACTOR drew
I-ACTOR barrymore
O   from
O   the
B-YEAR  1980s

O   what
O   movies
O   starred
O   both
B-ACTOR al
I-ACTOR pacino

Это образец из моего набора данных, который не работает, 

O   dated
O   as
O   of
B-STARTDATE February
I-STARTDATE 9
I-STARTDATE ,
L-STARTDATE 2017
O   by
O   and
O   between
O   Allenware
O   Ltd

Я не могу определить проблему, пожалуйстаhelp.

Обновление

добавление stderr.log по запросу. 

0it [00:00, ?it/s]
1it [00:00, 556.72it/s]

0it [00:00, ?it/s]
Traceback (most recent call last):
  File "/allennlp/bin/allennlp", line 8, in <module>
    sys.exit(run())
  File "/allennlp/lib/python3.6/site-packages/allennlp/run.py", line 18, in run
    main(prog="allennlp")
  File "/allennlp/lib/python3.6/site-packages/allennlp/commands/__init__.py", line 102, in main
    args.func(args)
  File "/allennlp/lib/python3.6/site-packages/allennlp/commands/train.py", line 124, in train_model_from_args
    args.cache_prefix)
  File "/allennlp/lib/python3.6/site-packages/allennlp/commands/train.py", line 168, in train_model_from_file
    cache_directory, cache_prefix)
  File "/allennlp/lib/python3.6/site-packages/allennlp/commands/train.py", line 226, in train_model
    cache_prefix)
  File "/allennlp/lib/python3.6/site-packages/allennlp/training/trainer_pieces.py", line 42, in from_params
    all_datasets = training_util.datasets_from_params(params, cache_directory, cache_prefix)
  File "/allennlp/lib/python3.6/site-packages/allennlp/training/util.py", line 185, in datasets_from_params
    validation_data = validation_and_test_dataset_reader.read(validation_data_path)
  File "/allennlp/lib/python3.6/site-packages/allennlp/data/dataset_readers/dataset_reader.py", line 134, in read
    instances = [instance for instance in Tqdm.tqdm(instances)]
  File "/allennlp/lib/python3.6/site-packages/allennlp/data/dataset_readers/dataset_reader.py", line 134, in <listcomp>
    instances = [instance for instance in Tqdm.tqdm(instances)]
  File "/allennlp/lib/python3.6/site-packages/tqdm/std.py", line 1081, in __iter__
    for obj in iterable:
  File "/allennlp/lib/python3.6/site-packages/allennlp/data/dataset_readers/conll2003.py", line 119, in _read
    ner_tags,tokens_ = fields
ValueError: not enough values to unpack (expected 2, got 1)
0it [00:00, ?it/s]

Добавление функций _read и text_to_instance 

@overrides
    def _read(self, file_path: str) -> Iterable[Instance]:
        # if `file_path` is a URL, redirect to the cache
        file_path = cached_path(file_path)

        with open(file_path, "r") as data_file:
            logger.info("Reading instances from lines in file at: %s", file_path)

            # Group into alternative divider / sentence chunks.
            for is_divider, lines in itertools.groupby(data_file, _is_divider):
                # Ignore the divider chunks, so that `lines` corresponds to the words
                # of a single sentence.
                if not is_divider:
                    fields = [line.strip().split() for line in lines]
                    # unzipping trick returns tuples, but our Fields need lists
                    fields = [list(field) for field in zip(*fields)]
                    ner_tags,tokens_ = fields
                    # TextField requires ``Token`` objects
                    tokens = [Token(token) for token in tokens_]

                    yield self.text_to_instance(tokens,ner_tags)

    def text_to_instance(  # type: ignore
        self,
        tokens: List[Token],
        ner_tags: List[str] = None,
    ) -> Instance:
        """
        We take `pre-tokenized` input here, because we don't have a tokenizer in this class.
        """

        sequence = TextField(tokens, self._token_indexers)
        instance_fields: Dict[str, Field] = {"tokens": sequence}
        instance_fields["metadata"] = MetadataField({"words": [x.text for x in tokens]})
        coded_ner=ner_tags
        if 'ner' in self.feature_labels:
            if coded_ner is None:
                raise ConfigurationError("Dataset reader was specified to use NER tags as "
                                         " features. Pass them to text_to_instance.")
            instance_fields['ner_tags'] = SequenceLabelField(coded_ner, sequence, "ner_tags")
        if self.tag_label == 'ner' and coded_ner is not None:
            instance_fields['tags'] = SequenceLabelField(coded_ner, sequence,self.label_namespace)
        return Instance(instance_fields)
...