Добавление TEZ в среду Hadoop / Hive / Spark делает Hive неработоспособным - PullRequest
Новая
       27

Добавление TEZ в среду Hadoop / Hive / Spark делает Hive неработоспособным

0 голосов
/ 13 марта 2019

У меня есть рабочий Hive on spark (загружается из Интернета в виде отдельных дистрибутивов):

  • Hive 2.3.0
  • Spark 2.3.2
  • Hadoop 2.8.5

Однако, когда я добавляю локально скомпилированную TEZ (0.9.1), ни функцию hive-on-tez, ни функцию hive-on-spark.

Этапы добавления TEZ:

  1. Компиляция mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true
  2. Создание папки /home/sergey/tez с подкаталогами tez и conf
  3. Распаковка tez-minimal.tar.gz в ~/tez/tez

  4. Добавьте tez-site.xml к ~/tez/conf/ с содержанием:

    <configuration> <property>
    <name>tez.lib.uris</name>
    <value>hdfs://localhost:9000/user/tez/tez-0.9.1.tar.gz</value>
    </property>
    </configuration>

  5. Перемещение tez-0.9.1.tar.gz в hdfs:///user/tez

  6. Обновление hadoop-env.sh с: 
    export TEZ_CONF_DIR=/home/sergey/tez/conf/
    export TEZ_JARS=/home/sergey/tez/tez/
    export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*:${HADOOP_CLASSPATH}:${JAVA_JDBC_LIBS}:${MAPREDUCE_LIBS}

Что касается Hive-On-Spark, ошибки в журнале Hive: 

2019-03-13T10:59:57,404 ERROR [802e96a0-867a-4bb1-8f86-d17e67d17802 main] spark.SparkTask: Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)'
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.

Caused by: java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s signer information does not match signer information of other classes in the same package

Что касается Hive-on-Tez: 

2019-03-13T11:20:01,626  INFO [7e669e87-40ed-4837-855a-3a0c98bbd441 main] client.TezClient: App did not succeed. Diagnostics: Application application_1552463853250_0002 failed 2 times due to AM Container for appattempt_1552463853250_0002_000002 exited with  exitCode: 1
Failing this attempt.Diagnostics: Exception from container-launch.
Container id: container_1552463853250_0002_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
    at org.apache.hadoop.util.Shell.run(Shell.java:869)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:235)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

И ошибка отyarn logs applicationID для hive-on-tez: ​​

2019-03-13 12:11:24,642 [ERROR] [main] |app.DAGAppMaster|: Error starting DAGAppMaster
java.lang.IllegalAccessError: tried to access field com.google.protobuf.AbstractMessage.memoizedSize from class org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto
    at org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.getSerializedSize(DAGProtos.java:20515)
    at com.google.protobuf.AbstractMessageLite.writeTo(AbstractMessageLite.java:75)
    at org.apache.tez.common.TezUtils.writeConfInPB(TezUtils.java:148)
    at org.apache.tez.common.TezUtils.createByteStringFromConf(TezUtils.java:82)
    at org.apache.tez.common.TezUtils.createUserPayloadFromConf(TezUtils.java:100)
    at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:444)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2662)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
    at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2659)
    at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2464)
2019-03-13 12:11:24,645 [INFO] [pool-1-thread-1] |app.DAGAppMaster|: DAGAppMasterShutdownHook invoked
2019-03-13 12:11:24,647 [INFO] [pool-1-thread-1] |service.AbstractService|: Service org.apache.tez.dag.app.DAGAppMaster failed in state STOPPED; cause: java.lang.NullPointerException

Что я делаю неправильно, если добавляю TEZ?

...