Мы пытаемся выполнить локальную искровую отправку на YARN, существующую на другом сервере:
Идея в том, что мы пытаемся экстернализировать отправку из jupyterlab, существующего в облачном контейнере, в jupyterlb в местной среде; мы экспортировали те же файлы конфигурации, которые существовали в контейнере jupyterlab, в локальную среду (в нашем случае это ноутбук, с vpn, чтобы разрешить доступ).
Проблема в облачном контейнере заключается в отправке на пряжу get выполнено успешно.
2020-03-18 12:45:15 INFO RequestHedgingRMFailoverProxyProvider:89 - Created wrapped proxy for [rm1, rm2]
2020-03-18 12:45:15 INFO AHSProxy:42 - Connecting to Application History server at server_address
2020-03-18 12:45:15 INFO RequestHedgingRMFailoverProxyProvider:147 - Looking for the active RM in [rm1, rm2]...
2020-03-18 12:45:15 INFO RequestHedgingRMFailoverProxyProvider:171 - Found active RM [rm2]
2020-03-18 12:45:15 INFO Client:54 - Requesting a new application from cluster with 348 NodeManagers
2020-03-18 12:45:15 INFO Configuration:2752 - resource-types.xml not found
2020-03-18 12:45:15 INFO ResourceUtils:418 - Unable to find 'resource-types.xml'.
2020-03-18 12:45:15 INFO Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (346687 MB per container)
2020-03-18 12:45:15 INFO Client:54 - Will allocate AM container, with 61952 MB memory including 5632 MB overhead
2020-03-18 12:45:15 INFO Client:54 - Setting up container launch context for our AM
2020-03-18 12:45:15 INFO Client:54 - Setting up the launch environment for our AM container
2020-03-18 12:45:15 INFO Client:54 - Preparing resources for our AM container
2020-03-18 12:45:15 INFO HadoopFSDelegationTokenProvider:54 - getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1993770491_1, ugi=id(auth:KERBEROS)]]
2020-03-18 12:45:15 INFO DFSClient:703 - Created token for owner: HDFS_DELEGATION_TOKEN owner=id, renewer=yarn, realUser=, issueDate=1584535515764, maxDate=1585140315764, sequenceNumber=31968916, masterKeyId=874 on ha-hdfs:edwbiprdmil
2020-03-18 12:45:16 INFO metastore:377 - Trying to connect to metastore with URI server_uri
2020-03-18 12:45:16 INFO metastore:473 - Connected to metastore.
Итак, в конце концов, мы подключаемся к метастазу.
Теперь мы выполняем тот же сценарий на нашей локальной машине, и после нескольких настроек
!spark-submit --conf spark.authenticate=True --conf spark.sql.hive.manageFilesourcePartitions=False \
--name test_spark --master yarn --deploy-mode cluster --driver-memory 55g --num-executors 20 \
--executor-memory 24g --executor-cores 30 --queue ADHOC_TACTICAL --conf spark.dynamicAllocation.enabled=true \
--conf spark.shuffle.service.enabled=true --conf spark.dynamicAllocation.maxExecutors=100 \
--conf spark.dynamicAllocation.minExecutors=10 --conf spark.dynamicAllocation.executorIdleTimeout=240s \
--conf spark.yarn.tags=tag submit.py
и результат выполнения выглядит следующим образом:
2020-03-18 13:08:48,677] WARN Unable to load native-hadoop library for your platform... using builtin-java classes where applicable (org.apache.hadoop.util.NativeCodeLoader)
[2020-03-18 13:08:51,835] WARN The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. (org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory)
[2020-03-18 13:08:52,590] INFO Timeline service address: server/ws/v1/timeline/ (org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl)
[2020-03-18 13:09:22,519] INFO Failing over to rm2 (org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider)
[2020-03-18 13:09:48,026] INFO Exception while invoking getClusterMetrics of class ApplicationClientProtocolPBClientImpl over rm2 after 1 fail over attempts. Trying to fail over after sleeping for 36102ms. (org.apache.hadoop.io.retry.RetryInvocationHandler)
org.apache.hadoop.net.ConnectTimeoutException: Call From local to server failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=server:8032]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:751)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy30.getClusterMetrics(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:206)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy31.getClusterMetrics(Unknown Source)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:487)
at org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication$1.apply(Client.scala:165)
at org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication$1.apply(Client.scala:165)
at org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
at org.apache.spark.deploy.yarn.Client.logInfo(Client.scala:60)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:164)
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1135)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1527)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=dcmipphm13002.edc.nam.gm.com/10.125.2.20:8032]
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
... 26 more
Итак, есть мысли? Мы подумали о брандмауэре, который останавливает нас от соединения, или требуется разрешение, позволяющее серверу принимать запросы от локальных компьютеров, но сталкивался ли кто-либо из вас с той же проблемой с той же проблемой?