Установите Cloudera CDH6.2, кодирующую pySpark, в записную книжку Хюэ и запустите - PullRequest
Новая
       25

Установите Cloudera CDH6.2, кодирующую pySpark, в записную книжку Хюэ и запустите

1 голос
/ 09 мая 2019

Я установил cdh6.2 с последующими офисными документами, и он работает правильно.Но я все еще хочу написать код pySpark и запустить его в оттенке вместо кодирования pySpark в pySpark-shell.Я перепробовал много конфигураций на странице конфигурации тона CHD.что я сделал как: 

[notebook]

  [[interpreters]]

    [[[spark]]]
        name=Scala
        interface=livy
    [[[pyspark]]]
        name=PySpark
        interface=livy
    [[[r]]]
        name=R
        interface=livy

установлен ливи с версией 0.6.0, журнал сервера ливи: 


19/05/09 18:01:25 INFO server.AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: ;other allowed users: *
19/05/09 18:01:25 INFO utils.LineBufferedStream: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark) overrides detected (/opt/cloudera/parcels/CDH/lib/spark).
19/05/09 18:01:25 INFO utils.LineBufferedStream: WARNING: Running spark-class from user-defined location.
19/05/09 18:01:26 INFO utils.LineBufferedStream: Welcome to
19/05/09 18:01:26 INFO utils.LineBufferedStream:       ____              __
19/05/09 18:01:26 INFO utils.LineBufferedStream:      / __/__  ___ _____/ /__
19/05/09 18:01:26 INFO utils.LineBufferedStream:     _\ \/ _ \/ _ `/ __/  '_/
19/05/09 18:01:26 INFO utils.LineBufferedStream:    /___/ .__/\_,_/_/ /_/\_\   version 2.4.0-cdh6.2.0
19/05/09 18:01:26 INFO utils.LineBufferedStream:       /_/
19/05/09 18:01:26 INFO utils.LineBufferedStream:                         
19/05/09 18:01:26 INFO utils.LineBufferedStream: Using Scala version 2.11.12, OpenJDK 64-Bit Server VM, 1.8.0_212
19/05/09 18:01:26 INFO utils.LineBufferedStream: Branch HEAD
19/05/09 18:01:26 INFO utils.LineBufferedStream: Compiled by user jenkins on 2019-03-14T07:01:24Z
19/05/09 18:01:26 INFO utils.LineBufferedStream: Revision HEAD
19/05/09 18:01:26 INFO utils.LineBufferedStream: Url 
19/05/09 18:01:26 INFO utils.LineBufferedStream: Type --help for more information.
19/05/09 18:01:26 INFO recovery.StateStore$: Using BlackholeStateStore for recovery.
19/05/09 18:01:26 INFO sessions.BatchSessionManager: Recovered 0 batch sessions. Next session id: 0
19/05/09 18:01:26 INFO sessions.InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0
19/05/09 18:01:26 INFO sessions.InteractiveSessionManager: Heartbeat watchdog thread started.
19/05/09 18:01:26 INFO util.log: Logging initialized @1490ms
19/05/09 18:01:26 INFO server.Server: jetty-9.3.24.v20180605, build timestamp: 2018-06-06T01:11:56+08:00, git hash: 84205aa28f11a4f31f2a3b86d1bba2cc8ab69827
19/05/09 18:01:26 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@1522d8a0{/,file:///opt/apache-livy-0.6.0-incubating-bin/src/main/org/apache/livy/server,AVAILABLE}
19/05/09 18:01:26 INFO server.AbstractNCSARequestLog: Opened /opt/apache-livy-0.6.0-incubating-bin/logs/2019_05_09.request.log
19/05/09 18:01:26 INFO server.AbstractConnector: Started ServerConnector@77a98a6a{HTTP/1.1,[http/1.1]}{0.0.0.0:8998}
19/05/09 18:01:26 INFO server.Server: Started @1674ms
19/05/09 18:01:26 INFO server.WebServer: Starting server on http://cdh.n1.com:8998

1 Ответ

1 голос
/ 09 мая 2019

Вы можете увидеть после добавления: 

[desktop]
app_blacklist=

[notebook]
  show_notebooks=true
...