PySpark 2.4.4 получил ошибку JNI с i python 3.7

Question

Я загружаю последнюю версию Spark и немного изменил ее в конфигурации.

изменение в spark-env. sh:

PYSPARK_PYTHON=/data/software/miniconda3/bin/ipython

Когда я запускаю pyspark, оно поднялось ошибка, как показано ниже. Журналы ошибок:

Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.6.1 -- An enhanced Interactive Python. Type '?' for help.
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 7 more
[TerminalIPythonApp] WARNING | Unknown error in handling PYTHONSTARTUP file /data/software/spark-2.4.4-bin-without-hadoop/python/pyspark/shell.py:
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
/data/software/miniconda3/lib/python3.7/site-packages/IPython/core/shellapp.py in _exec_file(self, fname, shell_futures)
    338                                                  self.shell.user_ns,
    339                                                  shell_futures=shell_futures,
--> 340                                                  raise_exceptions=True)
    341         finally:
    342             sys.argv = save_argv

/data/software/miniconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py in safe_execfile(self, fname, exit_ignore, raise_exceptions, shell_futures, *where)
   2716                 py3compat.execfile(
   2717                     fname, glob, loc,
-> 2718                     self.compile if shell_futures else None)
   2719             except SystemExit as status:
   2720                 # If the call was made with 0 or None exit status (sys.exit(0)

/data/software/miniconda3/lib/python3.7/site-packages/IPython/utils/py3compat.py in execfile(fname, glob, loc, compiler)
    186     with open(fname, 'rb') as f:
    187         compiler = compiler or compile
--> 188         exec(compiler(f.read(), fname, 'exec'), glob, loc)
    189 
    190 # Refactor print statements in doctests.

/data/software/spark-2.4.4-bin-without-hadoop/python/pyspark/shell.py in <module>
     36     SparkContext.setSystemProperty("spark.executor.uri", os.environ["SPARK_EXECUTOR_URI"])
     37 
---> 38 SparkContext._ensure_initialized()
     39 
     40 try:

/data/software/spark-2.4.4-bin-without-hadoop/python/pyspark/context.py in _ensure_initialized(cls, instance, gateway, conf)
    314         with SparkContext._lock:
    315             if not SparkContext._gateway:
--> 316                 SparkContext._gateway = gateway or launch_gateway(conf)
    317                 SparkContext._jvm = SparkContext._gateway.jvm
    318 

/data/software/spark-2.4.4-bin-without-hadoop/python/pyspark/java_gateway.py in launch_gateway(conf)
     44     :return: a JVM gateway
     45     """
---> 46     return _launch_gateway(conf)
     47 
     48 

/data/software/spark-2.4.4-bin-without-hadoop/python/pyspark/java_gateway.py in _launch_gateway(conf, insecure)
    106 
    107             if not os.path.isfile(conn_info_file):
--> 108                 raise Exception("Java gateway process exited before sending its port number")
    109 
    110             with open(conn_info_file, "rb") as info:

Exception: Java gateway process exited before sending its port number

In [1]: exit                                                                                                                                                                                    
dennis@device2:/data/software/spark-2.4.4-bin-without-hadoop/conf$ java -version
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
dennis@device2:/data/software/spark-2.4.4-bin-without-hadoop/conf$ export PYSPARK_SUBMIT_ARGS="--master local[2] pyspark-shell"
dennis@device2:/data/software/spark-2.4.4-bin-without-hadoop/conf$ pyspark 
Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.6.1 -- An enhanced Interactive Python. Type '?' for help.
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 7 more
[TerminalIPythonApp] WARNING | Unknown error in handling PYTHONSTARTUP file /data/software/spark-2.4.4-bin-without-hadoop/python/pyspark/shell.py:
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
/data/software/miniconda3/lib/python3.7/site-packages/IPython/core/shellapp.py in _exec_file(self, fname, shell_futures)
    338                                                  self.shell.user_ns,
    339                                                  shell_futures=shell_futures,
--> 340                                                  raise_exceptions=True)
    341         finally:
    342             sys.argv = save_argv

/data/software/miniconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py in safe_execfile(self, fname, exit_ignore, raise_exceptions, shell_futures, *where)
   2716                 py3compat.execfile(
   2717                     fname, glob, loc,
-> 2718                     self.compile if shell_futures else None)
   2719             except SystemExit as status:
   2720                 # If the call was made with 0 or None exit status (sys.exit(0)

/data/software/miniconda3/lib/python3.7/site-packages/IPython/utils/py3compat.py in execfile(fname, glob, loc, compiler)
    186     with open(fname, 'rb') as f:
    187         compiler = compiler or compile
--> 188         exec(compiler(f.read(), fname, 'exec'), glob, loc)
    189 
    190 # Refactor print statements in doctests.

/data/software/spark-2.4.4-bin-without-hadoop/python/pyspark/shell.py in <module>
     36     SparkContext.setSystemProperty("spark.executor.uri", os.environ["SPARK_EXECUTOR_URI"])
     37 
---> 38 SparkContext._ensure_initialized()
     39 
     40 try:

/data/software/spark-2.4.4-bin-without-hadoop/python/pyspark/context.py in _ensure_initialized(cls, instance, gateway, conf)
    314         with SparkContext._lock:
    315             if not SparkContext._gateway:
--> 316                 SparkContext._gateway = gateway or launch_gateway(conf)
    317                 SparkContext._jvm = SparkContext._gateway.jvm
    318 

/data/software/spark-2.4.4-bin-without-hadoop/python/pyspark/java_gateway.py in launch_gateway(conf)
     44     :return: a JVM gateway
     45     """
---> 46     return _launch_gateway(conf)
     47 
     48 

/data/software/spark-2.4.4-bin-without-hadoop/python/pyspark/java_gateway.py in _launch_gateway(conf, insecure)
    106 
    107             if not os.path.isfile(conn_info_file):
--> 108                 raise Exception("Java gateway process exited before sending its port number")
    109 
    110             with open(conn_info_file, "rb") as info:

Exception: Java gateway process exited before sending its port number

Среда:

Java:

java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

Версия Spark: park-2.4.4-bin-without-hadoop

The Had oop это 3.0.0 (CDH-6.2.0)

Answer 1

Прежде всего, исключение произошло не из-за ipython 3.7, потому что Spark не может найти этот класс org.slf4j.Logger в пути к классам, когда инициализируется SparkContext (во время запуска pyspark в в данном случае).

Согласно вашему описанию, вы используете сборку Spark "has oop free", в то время как Spark зависит от Hadoop, поэтому вам нужно явно указать Spark, где взять Had oop пакет jars в соответствии с документацией Spark здесь: https://spark.apache.org/docs/latest/hadoop-provided.html и я думаю, что класс, который мы упомянули выше, каким-то образом связан с этими банками, поэтому Spark не смог его найти.

Вы можете попробовать два решения:

1. попытаться обновить SPARK_DIST_CLASSPATH в spark-env.sh, чтобы явным образом сообщить Spark, где найти банки, имеющие oop, если на вашем компьютере было oop.

2. попробуйте использовать эту сборку spark-2.4.4-bin-hadoop2.7.tgz на случай, если на вашем компьютере не было oop. В этой сборке Jar-файлы, связанные с oop, уже поставлены вместе со Spark, поэтому вам не стоит беспокоиться об этой проблеме.