При попытке использовать pyspark с Livy я получаю ошибку PYSPARK_GATEWAY_SECRET - PullRequest
Новая
       16

При попытке использовать pyspark с Livy я получаю ошибку PYSPARK_GATEWAY_SECRET

0 голосов
/ 12 сентября 2018

При запуске pyspark в командной строке с использованием pyspark все работает как положено. Однако при использовании Livy это не так.

Я установил соединение с помощью Почтальона. Сначала я отправлю это в конечную точку sessions: 

{
  "kind": "pyspark",
  "proxyUser": "spark"
}

Сессия раскручивается, я вижу, как Spark начинает работать на YARN. Тем не менее, я получаю эту ошибку в моем журнале контейнера: 

18/09/12 15:53:00 ERROR repl.PythonInterpreter: Process has died with 1
18/09/12 15:53:00 ERROR repl.PythonInterpreter: Traceback (most recent call last):
  File "/yarn/nm/usercache/livy/appcache/application_1535188013308_0051/container_1535188013308_0051_01_000001/tmp/3015653701235928503", line 643, in <module>
    sys.exit(main())
  File "/yarn/nm/usercache/livy/appcache/application_1535188013308_0051/container_1535188013308_0051_01_000001/tmp/3015653701235928503", line 533, in main
    exec('from pyspark.shell import sc', global_dict)
  File "<string>", line 1, in <module>
  File "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/python/lib/pyspark.zip/pyspark/shell.py", line 38, in <module>
  File "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 292, in _ensure_initialized
  File "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/python/lib/pyspark.zip/pyspark/java_gateway.py", line 47, in launch_gateway
  File "/usr/lib64/python2.7/UserDict.py", line 23, in __getitem__
    raise KeyError(key)
KeyError: 'PYSPARK_GATEWAY_SECRET'

Выходные данные sessions/XYZ/log: 

{
    "id": 16,
    "from": 0,
    "total": 46,
    "log": [
        "stdout: ",
        "\nstderr: ",
        "Warning: Skip remote jar hdfs://master1.lama.nuc:8020/livy/rsc/livy-api-0.4.0-SNAPSHOT.jar.",
        "Warning: Skip remote jar hdfs://master1.lama.nuc:8020/livy/rsc/livy-rsc-0.4.0-SNAPSHOT.jar.",
        "Warning: Skip remote jar hdfs://master1.lama.nuc:8020/livy/rsc/netty-all-4.0.29.Final.jar.",
        "Warning: Skip remote jar hdfs://master1.lama.nuc:8020/livy/repl/commons-codec-1.9.jar.",
        "Warning: Skip remote jar hdfs://master1.lama.nuc:8020/livy/repl/livy-core_2.11-0.4.0-SNAPSHOT.jar.",
        "Warning: Skip remote jar hdfs://master1.lama.nuc:8020/livy/repl/livy-repl_2.11-0.4.0-SNAPSHOT.jar.",
        "Warning: Skip remote jar hdfs://master1.lama.nuc:8020/lama/lama.main-assembly-0.9.0-spark2.3.0-hadoop2.6.5-SNAPSHOT.jar.",
        "18/09/12 15:52:50 INFO client.RMProxy: Connecting to ResourceManager at master1.lama.nuc/192.168.42.100:8032",
        "18/09/12 15:52:51 INFO yarn.Client: Requesting a new application from cluster with 6 NodeManagers",
        "18/09/12 15:52:51 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (12288 MB per container)",
        "18/09/12 15:52:51 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead",
        "18/09/12 15:52:51 INFO yarn.Client: Setting up container launch context for our AM",
        "18/09/12 15:52:51 INFO yarn.Client: Setting up the launch environment for our AM container",
        "18/09/12 15:52:51 INFO yarn.Client: Preparing resources for our AM container",
        "18/09/12 15:52:51 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://master1.lama.nuc:8020/livy/rsc/livy-api-0.4.0-SNAPSHOT.jar",
        "18/09/12 15:52:52 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://master1.lama.nuc:8020/livy/rsc/livy-rsc-0.4.0-SNAPSHOT.jar",
        "18/09/12 15:52:52 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://master1.lama.nuc:8020/livy/rsc/netty-all-4.0.29.Final.jar",
        "18/09/12 15:52:52 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://master1.lama.nuc:8020/livy/repl/commons-codec-1.9.jar",
        "18/09/12 15:52:52 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://master1.lama.nuc:8020/livy/repl/livy-core_2.11-0.4.0-SNAPSHOT.jar",
        "18/09/12 15:52:52 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://master1.lama.nuc:8020/livy/repl/livy-repl_2.11-0.4.0-SNAPSHOT.jar",
        "18/09/12 15:52:52 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://master1.lama.nuc:8020/lama/lama.main-assembly-0.9.0-spark2.3.0-hadoop2.6.5-SNAPSHOT.jar",
        "18/09/12 15:52:52 INFO yarn.Client: Uploading resource file:/tmp/spark-37413ebc-9427-44d8-8a01-c4222eb899f8/__spark_conf__7516701035111969209.zip -> hdfs://master1.lama.nuc:8020/user/livy/.sparkStaging/application_1535188013308_0051/__spark_conf__.zip",
        "18/09/12 15:52:53 INFO spark.SecurityManager: Changing view acls to: livy",
        "18/09/12 15:52:53 INFO spark.SecurityManager: Changing modify acls to: livy",
        "18/09/12 15:52:53 INFO spark.SecurityManager: Changing view acls groups to: ",
        "18/09/12 15:52:53 INFO spark.SecurityManager: Changing modify acls groups to: ",
        "18/09/12 15:52:53 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(livy); groups with view permissions: Set(); users  with modify permissions: Set(livy); groups with modify permissions: Set()",
        "18/09/12 15:52:57 INFO yarn.Client: Submitting application application_1535188013308_0051 to ResourceManager",
        "18/09/12 15:52:57 INFO impl.YarnClientImpl: Submitted application application_1535188013308_0051",
        "18/09/12 15:52:57 INFO yarn.Client: Application report for application_1535188013308_0051 (state: ACCEPTED)",
        "18/09/12 15:52:57 INFO yarn.Client: ",
        "\t client token: N/A",
        "\t diagnostics: N/A",
        "\t ApplicationMaster host: N/A",
        "\t ApplicationMaster RPC port: -1",
        "\t queue: root.users.livy",
        "\t start time: 1536760377659",
        "\t final status: UNDEFINED",
        "\t tracking URL: http://master1.lama.nuc:8088/proxy/application_1535188013308_0051/",
        "\t user: livy",
        "18/09/12 15:52:57 INFO util.ShutdownHookManager: Shutdown hook called",
        "18/09/12 15:52:57 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-795d9b05-a5ad-4930-ad8b-77034022bc17",
        "18/09/12 15:52:57 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-37413ebc-9427-44d8-8a01-c4222eb899f8",
        "\nYARN Diagnostics: "
    ]
}

Что здесь не так? Использование CDH 5.15.0 с Parcels и Spark2. Использование Scala работает без проблем.

Followup

Я установил режим развертывания от cluster до client. KeyError пропадает, но при попытке запустить даже простой sc.version я получаю Interpreter died без отслеживания или ошибки вообще.

1 Ответ

0 голосов
/ 13 октября 2018

Я столкнулся с той же проблемой и решил ее, обновившись до Livy 0.5.0.

Очевидно, что CDH 5.15.0 имеет исправление для уязвимости безопасности (CVE-2018-1334), которая внесла несовместимость с Livy <0.5.0. Автор благодарности Марсело Ванзину за размещение этого сообщения в архивах <a href="http://mail-archives.apache.org/mod_mbox/livy-user/201809.mbox/%3CCAAOnQ7tGLeL1X3rBmZ90A7dR=nx+u5Uk2-ivLNSXykZ6Yj9LCg@mail.gmail.com%3E" rel="nofollow noreferrer"> livy-user .

...