Я пытаюсь связать локальные библиотеки BLAS в простом примере, чтобы проверить разницу в производительности с «обычной» реализацией.Я покорно следовал инструкциям, опубликованным на официальном GitHub netlib-java, но все равно получаю ошибку, похожую на эту тему .
Я использую предварительно скомпилированный Spark 2.3 с Hadoop 2.7;Я попытался собрать Spark из исходного кода с возможностью явного включения netlib-java (как описано на странице Spark MLlib , но безрезультатно.
Конкретные сообщения об ошибках, которые меня смущают:выглядит так:
spark-shell --packages com.github.fommil.netlib:all:1.1.2
Ivy Default Cache set to: /home/user/.ivy2/cache
The jars for the packages stored in: /home/user/.ivy2/jars
:: loading settings :: url = jar:file:/home/user/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.github.fommil.netlib#all added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found com.github.fommil.netlib#all;1.1.2 in central
found net.sourceforge.f2j#arpack_combined_all;0.1 in central
found com.github.fommil.netlib#core;1.1.2 in central
found com.github.fommil.netlib#netlib-native_ref-osx-x86_64;1.1 in central
found com.github.fommil.netlib#native_ref-java;1.1 in central
found com.github.fommil#jniloader;1.1 in central
found com.github.fommil.netlib#netlib-native_ref-linux-x86_64;1.1 in central
found com.github.fommil.netlib#netlib-native_ref-linux-i686;1.1 in central
found com.github.fommil.netlib#netlib-native_ref-win-x86_64;1.1 in central
found com.github.fommil.netlib#netlib-native_ref-win-i686;1.1 in central
found com.github.fommil.netlib#netlib-native_ref-linux-armhf;1.1 in central
found com.github.fommil.netlib#netlib-native_system-osx-x86_64;1.1 in central
found com.github.fommil.netlib#native_system-java;1.1 in central
found com.github.fommil.netlib#netlib-native_system-linux-x86_64;1.1 in central
found com.github.fommil.netlib#netlib-native_system-linux-i686;1.1 in central
found com.github.fommil.netlib#netlib-native_system-linux-armhf;1.1 in central
found com.github.fommil.netlib#netlib-native_system-win-x86_64;1.1 in central
found com.github.fommil.netlib#netlib-native_system-win-i686;1.1 in central
:: resolution report :: resolve 1196ms :: artifacts dl 16ms
:: modules in use:
com.github.fommil#jniloader;1.1 from central in [default]
com.github.fommil.netlib#all;1.1.2 from central in [default]
com.github.fommil.netlib#core;1.1.2 from central in [default]
com.github.fommil.netlib#native_ref-java;1.1 from central in [default]
com.github.fommil.netlib#native_system-java;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_ref-linux-armhf;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_ref-linux-i686;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_ref-linux-x86_64;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_ref-osx-x86_64;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_ref-win-i686;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_ref-win-x86_64;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_system-linux-armhf;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_system-linux-i686;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_system-linux-x86_64;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_system-osx-x86_64;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_system-win-i686;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_system-win-x86_64;1.1 from central in [default]
net.sourceforge.f2j#arpack_combined_all;0.1 from central in [default]
:: evicted modules:
com.github.fommil.netlib#core;1.1 by [com.github.fommil.netlib#core;1.1.2] in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 19 | 0 | 0 | 1 || 17 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
confs: [default]
0 artifacts copied, 17 already retrieved (0kB/15ms)
2018-06-13 07:53:28 WARN Utils:66 - Your hostname, wdfl30003105a resolves to a loopback address: 127.0.0.1; using 10.18.91.86 instead (on interface enp1s0)
2018-06-13 07:53:28 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2018-06-13 07:53:30 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Warning: Local jar /home/user/.ivy2/jars/net.sourceforge.f2j_arpack_combined_all-0.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-osx-x86_64-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-linux-i686-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-win-x86_64-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-win-i686-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-linux-armhf-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_system-osx-x86_64-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_system-linux-x86_64-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_system-linux-i686-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_system-linux-armhf-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_system-win-x86_64-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_system-win-i686-1.1.jar does not exist, skipping.
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
2018-06-13 07:53:44 ERROR SparkContext:91 - Failed to add file:/home/user/.ivy2/jars/net.sourceforge.f2j_arpack_combined_all-0.1.jar to Spark environment
java.io.FileNotFoundException: Jar /home/user/.ivy2/jars/net.sourceforge.f2j_arpack_combined_all-0.1.jar not found
at org.apache.spark.SparkContext.addJarFile$1(SparkContext.scala:1807)
at org.apache.spark.SparkContext.addJar(SparkContext.scala:1837)
at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:457)
at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:457)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:457)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:103)
at $line3.$read$$iw$$iw.<init>(<console>:15)
at $line3.$read$$iw.<init>(<console>:43)
at $line3.$read.<init>(<console>:45)
at $line3.$read$.<init>(<console>:49)
at $line3.$read$.<clinit>(<console>)
at $line3.$eval$.$print$lzycompute(<console>:7)
at $line3.$eval$.$print(<console>:6)
at $line3.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(SparkILoop.scala:79)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(SparkILoop.scala:79)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SparkILoop.scala:79)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:79)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:79)
at scala.tools.nsc.interpreter.ILoop.savingReplayStack(ILoop.scala:91)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:78)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:78)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:78)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:77)
at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:110)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
at org.apache.spark.repl.Main$.doMain(Main.scala:76)
at org.apache.spark.repl.Main$.main(Main.scala:56)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2018-06-13 07:53:44 ERROR SparkContext:91 - Failed to add file:/home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-osx-x86_64-1.1.jar to Spark environment
java.io.FileNotFoundException: Jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-osx-x86_64-1.1.jar not found
at org.apache.spark.SparkContext.addJarFile$1(SparkContext.scala:1807)
at org.apache.spark.SparkContext.addJar(SparkContext.scala:1837)
at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:457)
at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:457)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:457)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:103)
at $line3.$read$$iw$$iw.<init>(<console>:15)
at $line3.$read$$iw.<init>(<console>:43)
at $line3.$read.<init>(<console>:45)
at $line3.$read$.<init>(<console>:49)
at $line3.$read$.<clinit>(<console>)
at $line3.$eval$.$print$lzycompute(<console>:7)
at $line3.$eval$.$print(<console>:6)
at $line3.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(SparkILoop.scala:79)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(SparkILoop.scala:79)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SparkILoop.scala:79)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:79)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:79)
at scala.tools.nsc.interpreter.ILoop.savingReplayStack(ILoop.scala:91)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:78)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:78)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:78)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:77)
at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:110)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
at org.apache.spark.repl.Main$.doMain(Main.scala:76)
at org.apache.spark.repl.Main$.main(Main.scala:56)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2018-06-13 07:53:44 ERROR SparkContext:91 - Failed to add file:/home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-linux-i686-1.1.jar to Spark environment
java.io.FileNotFoundException: Jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-linux-i686-1.1.jar not found
В основном это говорит о том, что библиотеки не могут быть найдены, но, что интересно, моя папка ~/.ivy2/jars показывает, что присутствуют все необходимые пакеты, за исключением немного другого имени:
~/.ivy2/jars$ ls
com.github.fommil_jniloader-1.1.jar
com.github.fommil.netlib_core-1.1.2.jar
com.github.fommil.netlib_native_ref-java-1.1.jar
com.github.fommil.netlib_native_system-java-1.1.jar
com.github.fommil.netlib_netlib-native_ref-linux-armhf-1.1-natives.jar
com.github.fommil.netlib_netlib-native_ref-linux-i686-1.1-natives.jar
com.github.fommil.netlib_netlib-native_ref-linux-x86_64-1.1.jar
com.github.fommil.netlib_netlib-native_ref-linux-x86_64-1.1-natives.jar
com.github.fommil.netlib_netlib-native_ref-osx-x86_64-1.1-natives.jar
com.github.fommil.netlib_netlib-native_ref-win-i686-1.1-natives.jar
com.github.fommil.netlib_netlib-native_ref-win-x86_64-1.1-natives.jar
com.github.fommil.netlib_netlib-native_system-linux-armhf-1.1-natives.jar
com.github.fommil.netlib_netlib-native_system-linux-i686-1.1-natives.jar
com.github.fommil.netlib_netlib-native_system-linux-x86_64-1.1-natives.jar
com.github.fommil.netlib_netlib-native_system-osx-x86_64-1.1-natives.jar
com.github.fommil.netlib_netlib-native_system-win-i686-1.1-natives.jar
com.github.fommil.netlib_netlib-native_system-win-x86_64-1.1-natives.jar
net.sourceforge.f2j_arpack_combined_all-0.1-javadoc.jar
Я недостаточно знаю об упаковке и / или о том, как Spark / Scala обрабатывает эти пакеты внутри. Для фиксированных пакетов в maven существует обходной путь, включающий PomOnly(),но, к сожалению, опция spark-shell --packages не позволяет мне делать нечто подобное.