У меня проблема с запуском Glue Jobs и получением ошибок. Я перехожу по этой ссылке в качестве ссылки.
Используемые банки:
s3: // Bucket / GlueJars / Snowkeke-JDB c - 3.9.1.jar,
s3: //bucket/GlueJars/spark-snowflake_2.12-2.5.9-spark_2.4.jar
Версия снежинки: выберите current_version ()
Что является причиной root?
Журналы ошибок Из клея:
py4j.protocol.Py4JJavaError: An error occurred while calling o75.load.
: java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V
at net.snowflake.spark.snowflake.Parameters$MergedParameters.<init>(Parameters.scala:263)
at net.snowflake.spark.snowflake.Parameters$.mergeParameters(Parameters.scala:257)
at net.snowflake.spark.snowflake.DefaultSource.createRelation(DefaultSource.scala:59)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
2020-02-22 12:55:29,258 ERROR [Driver] ya
rn.ApplicationMaster (Logging.scala:logError(70)) - User application exited with status 1
2020-02-22 12:55:29,258 INFO [Driver] yarn.ApplicationMaster (Logging.scala:logInfo(54)) - Final app status: FAILED, exitCode: 1, (reason: User application exited with status 1)
2020-02-22 12:55:29,261 INFO [pool-4-thread-1] spark.SparkContext (Logging.scala:logInfo(54)) - Invoking stop() from shutdown hook
2020-02-22 12:55:29,264 INFO [pool-4-thread-1] server.AbstractConnector (AbstractConnector.java:doStop(318)) - Stopped Spark@dc562d7{HTTP/1.1,[http/1.1]}{0.0.0.0:0}
2020-02-22 12:55:29,266 INFO [pool-4-thread-1] ui.SparkUI (Logging.scala:logInfo(54)) - Stopped Spark web UI at http://ip-172-32-153-220.ec2.internal:41151
2020-02-22 12:55:29,268 INFO [dispatcher-event-loop-1] yarn.YarnAllocator (Logging.scala:logInfo(54)) - Driver requested a total number of 0 executor(s).
2020-02-22 12:55:29,269 INFO [pool-4-thread-1] cluster.YarnClusterSchedulerBackend (Logging.scala:logInfo(54)) - Shutting down all executors
2020-02-22 12:55:29,269 INFO [dispatcher-event-loop-2] cluster.YarnSchedulerBackend$YarnDriverEndpoint (Logging.scala:logInfo(54)) - Asking each executor to shut down
2020-02-22 12:55:29,272 INFO [pool-4-thread-1] cluster.SchedulerExtensionServices (Logging.scala:logInfo(54)) - Stopping SchedulerExtensionServices
(serviceOption=None,
services=List(),
started=false)
2020-02-22 12:55:29,275 INFO [dispatcher-event-loop-2] spark.MapOutputTrackerMasterEndpoint (Logging.scala:logInfo(54)) - MapOutputTrackerMasterEndpoint stopped!
2020-02-22 12:55:29,284 INFO [pool-4-thread-1] memory.MemoryStore (Logging.scala:logInfo(54)) - MemoryStore cleared
2020-02-22 12:55:29,284 INFO [pool-4-thread-1] storage.BlockManager (Logging.scala:logInfo(54)) - BlockManager stopped
2020-02-22 12:55:29,285 INFO [pool-4-thread-1] storage.BlockManagerMaster (Logging.scala:logInfo(54)) - BlockManagerMaster stopped
2020-02-22 12:55:29,287 INFO [dispatcher-event-loop-3] scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint (Logging.scala:logInfo(54)) - OutputCommitCoordinator stopped!
2020-02-22 12:55:29,290 INFO [pool-4-thread-1] spark.SparkContext (Logging.scala:logInfo(54)) - Successfully stopped SparkContext
2020-02-22 12:55:29,291 INFO [pool-4-thread-1] yarn.ApplicationMaster (Logging.scala:logInfo(54)) - Unregistering ApplicationMaster with FAILED (diag message: User application exited with status 1)
2020-02-22 12:55:29,300 INFO [pool-4-thread-1] impl.AMRMClientImpl (AMRMClientImpl.java:unregisterApplicationMaster(476)) - Waiting for application to be successfully unregistered.
2020-02-22 12:55:29,401 INFO [pool-4-thread-1] yarn.ApplicationMaster (Logging.scala:logInfo(54)) - Deleting staging directory hdfs://ip-172-32-141-92.ec2.internal:8020/user/root/.sparkStaging/application_1582375951668_0001
2020-02-22 12:55:29,411 INFO [pool-4-thread-1] util.ShutdownHookManager (Logging.scala:logInfo(54)) - Shutdown hook called
2020-02-22 12:55:29,412 INFO [pool-4-thread-1] util.ShutdownHookManager (Logging.scala:logInfo(54)) - Deleting directory /mnt/yarn/usercache/root/appcache/application_1582375951668_0001/spark-9c440196-6577-41ee-bf58-dfc1dcb6726a
2020-02-22 12:55:29,414 INFO [pool-4-thread-1] util.ShutdownHookManager (Logging.scala:logInfo(54)) - Deleting directory /mnt/yarn/usercache/root/appcache/application_1582375951668_0001/spark-9c440196-6577-41ee-bf58-dfc1dcb6726a/pyspark-c95bbd70-903e-488a-8bda-15c23b4bfe4f
End of LogType:stdout