Spark job Issue [java.lang.VerifyError: класс com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer переопределяет конечный метод с помощью Resreolved] - PullRequest
Новая
       9

Spark job Issue [java.lang.VerifyError: класс com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer переопределяет конечный метод с помощью Resreolved]

0 голосов
/ 25 января 2019

Я пытаюсь запустить мой искровой код, но у меня проблема: 

Exception in thread "main" java.lang.VerifyError: class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides final method withResolved.(Lcom/fasterxml/jackson/databind/BeanProperty;Lcom/fasterxml/jackson/databind/jsontype/TypeSerializer;Lcom/fasterxml/jackson/databind/JsonSerializer;)Lcom/fasterxml/jackson/databind/ser/std/AsArraySerializerBase;
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.fasterxml.jackson.module.scala.ser.IteratorSerializerModule$class.$init$(IteratorSerializerModule.scala:70)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.<init>(DefaultScalaModule.scala:18)
at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<init>(DefaultScalaModule.scala:33)
at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<clinit>(DefaultScalaModule.scala)
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
at org.apache.spark.sql.Dataset.rdd$lzycompute(Dataset.scala:2586)
at org.apache.spark.sql.Dataset.rdd(Dataset.scala:2583)
at org.apache.spark.sql.Dataset.toJavaRDD(Dataset.scala:2596)

Вот мой пом: 

            <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.11.12</version>
        </dependency>

<dependency>
    <groupId>org.scala-lang</groupId>
    <artifactId>scala-reflect</artifactId>
    <version>2.11.12</version>
</dependency>

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.6.7.1</version>
</dependency>

<dependency>
    <groupId>com.fasterxml.jackson.module</groupId>
    <artifactId>jackson-module-scala_2.11</artifactId>
    <version>2.6.7</version>
</dependency>

<dependency>
    <groupId>com.fasterxml.jackson.module</groupId>
    <artifactId>jackson-module-paranamer</artifactId>
    <version>2.6.7</version>
</dependency>


<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-annotations</artifactId>
    <version>2.6.7</version>
</dependency>

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-core</artifactId>
    <version>2.6.7</version>
</dependency>

<dependency>
    <groupId>org.mariadb.jdbc</groupId>
    <artifactId>mariadb-java-client</artifactId>
    <version>2.3.0</version>
</dependency>

<dependency>
    <groupId>com.typesafe</groupId>
    <artifactId>config</artifactId>
    <version>1.3.3</version>
</dependency>

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>2.2.1</version>
    <exclusions>
        <exclusion>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>*</artifactId>
        </exclusion>

        <exclusion>
            <groupId>com.fasterxml.jackson.module</groupId>
            <artifactId>*</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.11</artifactId>
    <version>2.2.1</version>
</dependency>

Вот вывод дерева зависимостей maven, относящегося к fastxml (зависимость mvn: tree | grep -i com.fasterxml.jackson): 

[INFO] |  |  |  +- com.fasterxml.jackson:jackson-datatype-guava:jar:1.9.1:compile
[INFO] |  |  |  \- com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.4.5:compile
[INFO] |  |  |     \- com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.4.5:compile
[INFO] |  |  |  \- com.fasterxml.jackson.datatype:jackson-datatype-joda:jar:2.5.1:compile
[INFO] |  +- com.fasterxml:jackson-xml-databind:jar:0.6.2:compile
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.6.7.1:compile
[INFO] +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.6.7:compile
[INFO] +- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.6.7:compile
[INFO] +- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.7:compile
[INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.6.7:compile

Я пробовал еще несколько версий fastxml, но они также выводят то же исключение. Понижение версии, а также обновление версии не работает. Какими должны быть мои следующие шаги?

Обновление ::::

Я обновил версию до 2.9.2, но даже с этим я получаю ту же ошибку.

вот дерево зависимостей maven после этого

mvn зависимость: дерево | grep "com.fasterxml.jackson" 

[INFO] |  |  |  +- com.fasterxml.jackson:jackson-datatype-guava:jar:1.9.1:compile
[INFO] |  |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.4.5:compile
[INFO] |  |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-xml:jar:2.4.5:compile
[INFO] |  |  |  \- com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.4.5:compile
[INFO] |  |  |     \- com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.4.5:compile
[INFO] |  |  |  \- com.fasterxml.jackson.datatype:jackson-datatype-joda:jar:2.5.1:compile
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.2:compile
[INFO] +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.2:compile
[INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.9.2:compile
[INFO] +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.9.2:compile
[INFO] +- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.9.2:compile
...