Spark UDF Map Merge представляется зависимым от раздела - PullRequest
Новая
       35

Spark UDF Map Merge представляется зависимым от раздела

0 голосов
/ 31 марта 2020

У меня есть UDF, который принимает массив столбцов схемы Seq[Annotation], где (это аннотации SparkNLP):

https://github.com/JohnSnowLabs/spark-nlp/blob/master/src/main/scala/com/johnsnowlabs/nlp/Annotation.scala

case class Annotation(annotatorType: String,
                      begin: Int,
                      end: Int,
                      result: String,
                      metadata: Map[String, String],
                      embeddings: Array[Float] = Array.emptyFloatArray
                     )

Поле метаданных каждого Annotation содержит ключ chunk и значение, которое может быть преобразовано в int. Я использую это значение, чтобы вводить аннотации и возвращать их в виде Map следующим образом: 

def keyedResolvedEntityTest: UserDefinedFunction = {
    val f = (cols: Seq[Seq[Row]]) => {
      cols
        .flatMap(annotationSeq => annotationSeq.map(Annotation(_)))
        .filter { annotation =>
          annotation.result.nonEmpty && (annotation.metadata.get(SparkNLPMetaFields.chunk) match {
            case Some(chunk) => chunk.nonEmpty
            case _           => false
          })
        }
        .groupBy(annotation => annotation.metadata(SparkNLPMetaFields.chunk).toInt)
    }
    udf(f)
  }

Тем не менее, я получаю совершенно разные результаты, основанные на разбиении. Когда принудительно создается один раздел, все Annotations отображаются для каждого блока с правильным ключом. Тем не менее, когда я делю данные, многие ключи теряются.

При принудительном разбиении 1: 

+---+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|key|value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+---+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|69 |[[chunk, 3453, 3468, TEXT, [entity -> TREATMENT, sentence -> 43, chunk -> 69], []], [assertion, 3453, 3468, present, [sentence -> 43, chunk -> 69], []], [entity, 3453, 3468, 20114009, [chunk -> 69, all_k_results -> RESULTS, confidence -> 0.04245374784489365, all_k_resolutions -> RESULTS, target_text -> TEXT, confidence_ratio -> 1.0128860062128966, token -> TEXT, resolved_text -> flier, code_type -> TYPE, all_k_confidences -> 0.04245374784489365:|:0.04191364831233573:|:0.040636733655387926:|:0.040437590019252075:|:0.04039633698635584:|:0.04022755310524273:|:0.04022755310524273:|:0.04017561672815233:|:0.04004100443013706:|:0.040007871257287504:|:0.04000453669337517:|:0.03986878437498474:|:0.03980645312042596:|:0.03973971790456786:|:0.039676433497812386:|:0.03967203237098835:|:0.039626935106025794:|:0.039587136912446516:|:0.03951755333954083:|:0.0394270968601706:|:0.039367121933287966:|:0.03931817645153327:|:0.03931448082142163:|:0.039281373078745636:|:0.039274512090385655, alternative_confidence_ratios -> 1.0128860062128966:|:1.031422669641129:|:1.0049247157419876:|:1.0010212072671383:|:1.0041957282530096:|:1.0:|:1.0012927337853164:|:1.0033618611703443:|:1.000828166353478:|:1.000083354643947:|:1.0034049776164131:|:1.0015658580373945:|:1.0016793077398878:|:1.001595012484148:|:1.0001109377705402:|:1.001138045746962:|:1.001005331445598:|:1.00176082694968:|:1.0022942718732508:|:1.0015234775603934:|:1.0012448563532703:|:1.0000940017529019:|:1.0008428356770938:|:1.0001746931532642, distance -> 3.8557784933125054, sentence -> 43], []]]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |

Без принудительного разделения: (Аннотация типа «сущность» полностью отсутствует.) 

|key|value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
+---+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|69 |[[chunk, 3453, 3468, TEXT, [entity -> TREATMENT, sentence -> 43, chunk -> 69], []], [assertion, 3453, 3468, present, [sentence -> 43, chunk -> 69], []]]

Есть ли способ убедиться, что вызов groupBy не зависит от раздела? Похоже, что единственно возможный вариант - перераспределить до UDF.

...