Я пытаюсь запустить интерактивную сессию искры через ливи в керберизованном искровом кластере. Я просмотрел основные статьи и настроил службы. Сеанс искры запускается в режиме пряжи.
Проблема возникает, когда пряжа пытается запустить контейнер, происходит сбой при возникновении проблемы аутентификации Причины: org. apache .had oop .security.AccessControlException: Клиент не может пройти аутентификацию через: [TOKEN, KERBEROS].
Полные журналы:
Application with id 23 submitted by user livy
2020-04-07 18:56:49,786 INFO security.DelegationTokenRenewer (DelegationTokenRenewer.java:handleAppSubmitEvent(458)) - application_1586250870601_0023 found existing hdfs token Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.1.12:8020, Ident: (token for livy: HDFS_DELEGATION_TOKEN owner=livy, renewer=yarn, realUser=livy/bspc00c76e1751-004.app.org@APP.ORG, issueDate=1586266003257, maxDate=1586870803257, sequenceNumber=128, masterKeyId=18)
2020-04-07 18:56:49,841 INFO security.DelegationTokenRenewer (DelegationTokenRenewer.java:renewToken(635)) - Renewed delegation-token= [Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.1.12:8020, Ident: (token for livy: HDFS_DELEGATION_TOKEN owner=livy, renewer=yarn, realUser=livy/bspc00c76e1751-004.app.org@APP.ORG, issueDate=1586266003257, maxDate=1586870803257, sequenceNumber=128, masterKeyId=18);exp=1586352409824; apps=[application_1586250870601_0023]]
2020-04-07 18:56:49,961 INFO security.DelegationTokenRenewer (DelegationTokenRenewer.java:renewToken(635)) - Renewed delegation-token= [Kind: kms-dt, Service: kms://http@bspc00c76e1751-001.app.org;bspc00c76e1751-002.app.org:9292/kms, Ident: (kms-dt owner=livy, renewer=yarn, realUser=livy, issueDate=1586266003402, maxDate=1586870803402, sequenceNumber=106, masterKeyId=42);exp=1586352409923; apps=[application_1586250870601_0023]]
2020-04-07 18:56:49,961 WARN token.Token (Token.java:getRenewer(497)) - No TokenRenewer defined for token kind HIVE_DELEGATION_TOKEN
2020-04-07 18:56:49,961 INFO security.DelegationTokenRenewer (DelegationTokenRenewer.java:setTimerForTokenRenewal(613)) - Renew Kind: kms-dt, Service: kms://http@bspc00c76e1751-001.app.org;bspc00c76e1751-002.app.org:9292/kms, Ident: (kms-dt owner=livy, renewer=yarn, realUser=livy, issueDate=1586266003402, maxDate=1586870803402, sequenceNumber=106, masterKeyId=42);exp=1586352409923; apps=[application_1586250870601_0023] in 86399962 ms, appId = [application_1586250870601_0023]
2020-04-07 18:56:49,961 INFO security.DelegationTokenRenewer (DelegationTokenRenewer.java:setTimerForTokenRenewal(613)) - Renew Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.1.12:8020, Ident: (token for livy: HDFS_DELEGATION_TOKEN owner=livy, renewer=yarn, realUser=livy/bspc00c76e1751-004.app.org@APP.ORG, issueDate=1586266003257, maxDate=1586870803257, sequenceNumber=128, masterKeyId=18);exp=1586352409824; apps=[application_1586250870601_0023] in 86399863 ms, appId = [application_1586250870601_0023]
2020-04-07 18:56:49,962 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1259)) - Storing application with id application_1586250870601_0023
2020-04-07 18:56:49,963 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1586250870601_0023 State change from NEW to NEW_SAVING on event = START
2020-04-07 18:56:49,969 INFO recovery.RMStateStore (RMStateStore.java:transition(222)) - Storing info for app: application_1586250870601_0023
2020-04-07 18:56:49,991 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1586250870601_0023 State change from NEW_SAVING to SUBMITTED on event = APP_NEW_SAVED
2020-04-07 18:56:49,993 INFO capacity.ParentQueue (ParentQueue.java:addApplication(494)) - Application added - appId: application_1586250870601_0023 user: livy leaf-queue of parent: root #applications: 1
2020-04-07 18:56:49,993 INFO capacity.CapacityScheduler (CapacityScheduler.java:addApplication(990)) - Accepted application application_1586250870601_0023 from user: livy, in queue: default
2020-04-07 18:56:49,994 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1586250870601_0023 State change from SUBMITTED to ACCEPTED on event = APP_ACCEPTED
2020-04-07 18:56:49,994 INFO resourcemanager.ApplicationMasterService (ApplicationMasterService.java:registerAppAttempt(479)) - Registering app attempt : appattempt_1586250870601_0023_000001
2020-04-07 18:56:49,994 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1586250870601_0023_000001 State change from NEW to SUBMITTED on event = START
2020-04-07 18:56:49,995 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(911)) - Application application_1586250870601_0023 from user: livy activated in queue: default
2020-04-07 18:56:49,995 INFO capacity.LeafQueue (LeafQueue.java:addApplicationAttempt(941)) - Application added - appId: application_1586250870601_0023 user: livy, leaf-queue: default #user-pending-applications: 0 #user-active-applications: 1 #queue-pending-applications: 0 #queue-active-applications: 1
2020-04-07 18:56:49,996 INFO capacity.CapacityScheduler (CapacityScheduler.java:addApplicationAttempt(1036)) - Added Application Attempt appattempt_1586250870601_0023_000001 to scheduler from user livy in queue default
2020-04-07 18:56:50,000 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1586250870601_0023_000001 State change from SUBMITTED to SCHEDULED on event = ATTEMPT_ADDED
2020-04-07 18:56:50,037 INFO allocator.AbstractContainerAllocator (AbstractContainerAllocator.java:getCSAssignmentFromAllocateResult(129)) - assignedContainer application attempt=appattempt_1586250870601_0023_000001 container=null queue=default clusterResource=<memory:15360, vCores:3> type=OFF_SWITCH requestedPartition=
2020-04-07 18:56:50,043 INFO rmcontainer.RMContainerImpl (RMContainerImpl.java:handle(490)) - container_e25_1586250870601_0023_01_000001 Container Transitioned from NEW to ALLOCATED
2020-04-07 18:56:50,043 INFO fica.FiCaSchedulerNode (FiCaSchedulerNode.java:allocateContainer(169)) - Assigned container container_e25_1586250870601_0023_01_000001 of capacity <memory:1024, vCores:1> on host bspc00c76e1751-003.app.org:45454, which has 1 containers, <memory:1024, vCores:1> used and <memory:14336, vCores:2> available after allocation
2020-04-07 18:56:50,044 INFO capacity.ParentQueue (ParentQueue.java:apply(1336)) - assignedContainer queue=root usedCapacity=0.06666667 absoluteUsedCapacity=0.06666667 used=<memory:1024, vCores:1> cluster=<memory:15360, vCores:3>
2020-04-07 18:56:50,044 INFO capacity.CapacityScheduler (CapacityScheduler.java:tryCommit(2900)) - Allocation proposal accepted
2020-04-07 18:56:50,051 INFO security.NMTokenSecretManagerInRM (NMTokenSecretManagerInRM.java:createAndGetNMToken(200)) - Sending NMToken for nodeId : bspc00c76e1751-003.app.org:45454 for container : container_e25_1586250870601_0023_01_000001
2020-04-07 18:56:50,051 INFO rmcontainer.RMContainerImpl (RMContainerImpl.java:handle(490)) - container_e25_1586250870601_0023_01_000001 Container Transitioned from ALLOCATED to ACQUIRED
2020-04-07 18:56:50,052 INFO security.NMTokenSecretManagerInRM (NMTokenSecretManagerInRM.java:clearNodeSetForAttempt(146)) - Clear node set for appattempt_1586250870601_0023_000001
2020-04-07 18:56:50,052 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:storeAttempt(2213)) - Storing attempt: AppId: application_1586250870601_0023 AttemptId: appattempt_1586250870601_0023_000001 MasterContainer: Container: [ContainerId: container_e25_1586250870601_0023_01_000001, AllocationRequestId: -1, Version: 0, NodeId: bspc00c76e1751-003.app.org:45454, NodeHttpAddress: bspc00c76e1751-003.app.org:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.113.144.183:45454 }, ExecutionType: GUARANTEED, ]
2020-04-07 18:56:50,052 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1586250870601_0023_000001 State change from SCHEDULED to ALLOCATED_SAVING on event = CONTAINER_ALLOCATED
2020-04-07 18:56:50,077 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1586250870601_0023_000001 State change from ALLOCATED_SAVING to ALLOCATED on event = ATTEMPT_NEW_SAVED
2020-04-07 18:56:50,078 INFO amlauncher.AMLauncher (AMLauncher.java:run(307)) - Launching masterappattempt_1586250870601_0023_000001
2020-04-07 18:56:50,081 INFO amlauncher.AMLauncher (AMLauncher.java:launch(109)) - Setting up container Container: [ContainerId: container_e25_1586250870601_0023_01_000001, AllocationRequestId: -1, Version: 0, NodeId: bspc00c76e1751-003.app.org:45454, NodeHttpAddress: bspc00c76e1751-003.app.org:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.113.144.183:45454 }, ExecutionType: GUARANTEED, ] for AM appattempt_1586250870601_0023_000001
2020-04-07 18:56:50,081 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createAndGetAMRMToken(195)) - Create AMRMToken for ApplicationAttempt: appattempt_1586250870601_0023_000001
2020-04-07 18:56:50,081 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createPassword(307)) - Creating password for appattempt_1586250870601_0023_000001
2020-04-07 18:56:50,165 INFO amlauncher.AMLauncher (AMLauncher.java:launch(130)) - Done launching container Container: [ContainerId: container_e25_1586250870601_0023_01_000001, AllocationRequestId: -1, Version: 0, NodeId: bspc00c76e1751-003.app.org:45454, NodeHttpAddress: bspc00c76e1751-003.app.org:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.113.144.183:45454 }, ExecutionType: GUARANTEED, ] for AM appattempt_1586250870601_0023_000001
2020-04-07 18:56:50,165 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1586250870601_0023_000001 State change from ALLOCATED to LAUNCHED on event = LAUNCHED
2020-04-07 18:56:50,862 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:updateAppCollectorsMap(714)) - Update collector information for application application_1586250870601_0023 with new address: bspc00c76e1751-003.app.org:12768 timestamp: 1586250870601, 32
2020-04-07 18:56:50,864 INFO rmcontainer.RMContainerImpl (RMContainerImpl.java:handle(490)) - container_e25_1586250870601_0023_01_000001 Container Transitioned from ACQUIRED to RUNNING
2020-04-07 18:56:51,072 INFO destination.HDFSAuditDestination (HDFSAuditDestination.java:logJSON(179)) - Flushing HDFS audit. Event Size:96
2020-04-07 18:56:51,072 INFO destination.HDFSAuditDestination (HDFSAuditDestination.java:flush(190)) - Flush called. name=yarn.async.multi_dest.batch.hdfs
2020-04-07 18:56:51,087 INFO destination.HDFSAuditDestination (HDFSAuditDestination.java:flush(199)) - Flush HDFS audit logs completed.....
2020-04-07 18:56:51,875 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:updateAppCollectorsMap(714)) - Update collector information for application application_1586250870601_0023 with new address: bspc00c76e1751-003.app.org:12768 timestamp: 1586250870601, 33
2020-04-07 18:56:51,966 INFO rmcontainer.RMContainerImpl (RMContainerImpl.java:handle(490)) - container_e25_1586250870601_0023_01_000001 Container Transitioned from RUNNING to COMPLETED
2020-04-07 18:56:51,966 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:rememberTargetTransitionsAndStoreState(1412)) - Updating application attempt appattempt_1586250870601_0023_000001 with final state: FAILED, and exit status: -1000
2020-04-07 18:56:51,967 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1586250870601_0023_000001 State change from LAUNCHED to FINAL_SAVING on event = CONTAINER_FINISHED
2020-04-07 18:56:51,980 INFO resourcemanager.ApplicationMasterService (ApplicationMasterService.java:unregisterAttempt(496)) - Unregistering app attempt : appattempt_1586250870601_0023_000001
2020-04-07 18:56:51,980 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:applicationMasterFinished(124)) - Application finished, removing password for appattempt_1586250870601_0023_000001
2020-04-07 18:56:51,981 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1586250870601_0023_000001 State change from FINAL_SAVING to FAILED on event = ATTEMPT_UPDATE_SAVED
2020-04-07 18:56:51,981 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1534)) - Max app attempts is 1 for application_1586250870601_0023, preventing further attempts.
2020-04-07 18:56:51,981 INFO rmapp.RMAppImpl (RMAppImpl.java:rememberTargetTransitionsAndStoreState(1278)) - Updating application application_1586250870601_0023 with final state: FAILED
2020-04-07 18:56:51,981 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1586250870601_0023 State change from ACCEPTED to FINAL_SAVING on event = ATTEMPT_FAILED
2020-04-07 18:56:51,982 INFO recovery.RMStateStore (RMStateStore.java:transition(260)) - Updating info for app: application_1586250870601_0023
2020-04-07 18:56:51,986 INFO capacity.CapacityScheduler (CapacityScheduler.java:doneApplicationAttempt(1085)) - Application Attempt appattempt_1586250870601_0023_000001 is done. finalState=FAILED
2020-04-07 18:56:51,986 INFO scheduler.AppSchedulingInfo (AppSchedulingInfo.java:clearRequests(159)) - Application application_1586250870601_0023 requests cleared
2020-04-07 18:56:51,986 INFO capacity.LeafQueue (LeafQueue.java:removeApplicationAttempt(1003)) - Application removed - appId: application_1586250870601_0023 user: livy queue: default #user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0 #queue-active-applications: 0
2020-04-07 18:56:51,992 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1197)) - Application application_1586250870601_0023 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1586250870601_0023_000001 exited with exitCode: -1000
Failing this attempt.Diagnostics: (Client.java:1559)
at org.apache.hadoop.ipc.Client.call(Client.java:1390)
... 37 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:615)
at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:411)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:801)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:797)
at java.security.AccessController.doPrivileged(Native Method)