Ошибка при создании задания куста в oozie с менеджером рабочего процесса - PullRequest
Новая
       16

Ошибка при создании задания куста в oozie с менеджером рабочего процесса

0 голосов
/ 18 сентября 2018

Я пытался разными способами создать улей в oozie, я использую Hortonworks-Sandbox для разработки. Я следую инструкциям на сайте https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.0.0/bk_workflow-management/content/hive_parameters.html (и многим другим, но не повезло).

Мой файл workflow.xml: 

<?xml version="1.0" encoding="UTF-8" standalone="no"?><workflow-app xmlns="uri:oozie:workflow:0.5" name="test">
<start to="Create_external_table"/>
<action name="Create_external_table">
    <hive xmlns="uri:oozie:hive-action:0.6">
        <job-tracker>${resourceManager}</job-tracker>
        <name-node>${nameNode}</name-node>
        <job-xml>/user/oozie/workflows/hive-conf.xml</job-xml>
        <configuration>
            <property>
                <name>hive.metastore.uris</name>
                <value>thrift://sandbox-hdp.hortonworks.com:9083</value>
            </property>
        </configuration>
        <query>CREATE EXTERNAL TABLE IF NOT EXISTS behavior_data.log_files (Id STRING, PortalId int, CampaignId int,BannerAssetId int, BannerAssetFileId int, Timestamp int, EventType int, Price float,Msisdn STRING,UserId STRING,hitDate STRING, hitTime int) COMMENT "log_files" ROW FORMAT DELIMITED FIELDS TERMINATED BY "|" STORED AS TEXTFILE LOCATION "/user/root/log_files/flumefiles/20180911/13";</query>
        <file>/user/root/log_files/flumefiles/20180911/13</file>
    </hive>
    <ok to="end"/>
    <error to="kill"/>
</action>
<kill name="kill">
    <message>${wf:errorMessage(wf:lastErrorNode())}</message>
</kill>
<end name="end"/>

И конфигурации прилагаются ниже.

Ошибки, которые я получаю от ambari-workflow manager: 

    `-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@:start:] Start action [0000013-180914115041330-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-09-14 15:44:36,823  INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@:start:] [***0000013-180914115041330-oozie-oozi-W@:start:***]Action status=DONE
2018-09-14 15:44:36,823  INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@:start:] [***0000013-180914115041330-oozie-oozi-W@:start:***]Action updated in DB!
2018-09-14 15:44:36,888  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000013-180914115041330-oozie-oozi-W@:start:
2018-09-14 15:44:36,889  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000013-180914115041330-oozie-oozi-W
2018-09-14 15:44:36,918  INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] Start action [0000013-180914115041330-oozie-oozi-W@Create_external_table] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-09-14 15:44:39,662  INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] Trying to get job [job_1536925639400_0005], attempt [1]
2018-09-14 15:44:39,712  INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] checking action, hadoop job ID [job_1536925639400_0005] status [RUNNING]
2018-09-14 15:44:39,716  INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] [***0000013-180914115041330-oozie-oozi-W@Create_external_table***]Action status=RUNNING
2018-09-14 15:44:39,716  INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] [***0000013-180914115041330-oozie-oozi-W@Create_external_table***]Action updated in DB!
2018-09-14 15:44:39,720  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] No Notification URL is defined. Therefore nothing to notify for job 0000013-180914115041330-oozie-oozi-W@Create_external_table
2018-09-14 15:55:00,272  INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] Trying to get job [job_1536925639400_0005], attempt [1]
2018-09-14 15:55:00,295  INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] checking action, hadoop job ID [job_1536925639400_0005] status [RUNNING]
2018-09-14 16:06:00,600  INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] Trying to get job [job_1536925639400_0005], attempt [1]
2018-09-14 16:06:00,623  INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] checking action, hadoop job ID [job_1536925639400_0005] status [RUNNING]
2018-09-14 16:17:00,299  INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] Trying to get job [job_1536925639400_0005], attempt [1]
2018-09-14 16:17:00,319  INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] checking action, hadoop job ID [job_1536925639400_0005] status [RUNNING]`

А из файла /var/log/oozie/oozie-error.log:

    2018-09-14 16:12:57,397  WARN ParameterVerifier:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition
2018-09-14 16:12:57,408  WARN LiteWorkflowAppService:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://sandbox-hdp.hortonworks.com:8020/user/oozie/workfl
ows/workflow_otinanai/lib] does not exist                                                                                                                                                                          
2018-09-18 08:20:38,605  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.hosts]                                                           
2018-09-18 08:20:38,655  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.hosts]                                                              
2018-09-18 08:20:38,726  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.groups]                                                          
2018-09-18 08:20:38,846  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.AuthorizationService.security.enabled]                                                             
2018-09-18 08:20:38,892  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [hadoop.security.credential.provider.path]                                                                        
2018-09-18 08:20:38,942  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [credentialStoreClassPath]                                                                                        
2018-09-18 08:20:38,943  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.groups]                                                             
2018-09-18 08:20:39,035  WARN Services:523 - SERVER[] System ID [oozie-oozi] exceeds maximum length [10], trimming                                                                                                 
2018-09-18 08:20:42,131  WARN Services:523 - SERVER[] Previous services singleton active, destroying it                                                                                                            
2018-09-18 08:20:42,189  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.hosts]                                                           
2018-09-18 08:20:42,204  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.hosts]                                                              
2018-09-18 08:20:42,213  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.groups]                                                          
2018-09-18 08:20:42,214  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.AuthorizationService.security.enabled]                                                             
2018-09-18 08:20:42,214  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [hadoop.security.credential.provider.path]                                                                        
2018-09-18 08:20:42,214  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [credentialStoreClassPath]                                                                                        
2018-09-18 08:20:42,215  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.groups]                                                             
2018-09-18 08:20:42,216  WARN Services:523 - SERVER[] System ID [oozie-oozi] exceeds maximum length [10], trimming                                                                                                 
2018-09-18 08:20:45,892  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.hosts]                                                           
2018-09-18 08:20:45,895  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.hosts]                                                              
2018-09-18 08:20:45,902  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.groups]                                                          
2018-09-18 08:20:45,902  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.AuthorizationService.security.enabled]                                                             
2018-09-18 08:20:45,903  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [hadoop.security.credential.provider.path]                                                                        
2018-09-18 08:20:45,904  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [credentialStoreClassPath]                                                                                        
2018-09-18 08:20:45,904  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.groups]                                                             
2018-09-18 08:20:45,909  WARN Services:523 - SERVER[] System ID [oozie-oozi] exceeds maximum length [10], trimming                                                                                                 
2018-09-18 08:20:48,136  WARN Services:523 - SERVER[] Previous services singleton active, destroying it                                                                                                            
2018-09-18 08:20:48,359  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.hosts]                                                           
2018-09-18 08:20:48,391  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.hosts]                                                              
2018-09-18 08:20:48,445  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.groups]                                                          
2018-09-18 08:20:48,454  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.AuthorizationService.security.enabled]                                                             
2018-09-18 08:20:48,469  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [hadoop.security.credential.provider.path]                                                                        
2018-09-18 08:20:48,498  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [credentialStoreClassPath]                                                                                        
2018-09-18 08:20:48,500  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.groups]                                                             
2018-09-18 08:20:48,504  WARN Services:523 - SERVER[] System ID [oozie-oozi] exceeds maximum length [10], trimming                                                                                                 
2018-09-18 08:20:55,819  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] System property [oozie.https.keystore.pass] no defined in Oozie configuration, ignored                                
2018-09-18 08:20:55,821  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] System property [oozie.admin.port] no defined in Oozie configuration, ignored                                         
2018-09-18 08:20:55,822  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] System property [oozie.https.keystore.file] no defined in Oozie configuration, ignored                                
2018-09-18 08:20:55,828  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.hosts]                                
2018-09-18 08:20:55,831  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.hosts]                                   
2018-09-18 08:20:55,834  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.groups]                               
2018-09-18 08:20:55,835  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [oozie.service.AuthorizationService.security.enabled]                                  
2018-09-18 08:20:55,836  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [hadoop.security.credential.provider.path]                                             
2018-09-18 08:20:55,837  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [credentialStoreClassPath]                                                             
2018-09-18 08:20:55,838  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.groups]                                  
2018-09-18 08:20:55,844  WARN Services:523 - SERVER[sandbox-hdp.hortonworks.com] System ID [oozie-oozi] exceeds maximum length [10], trimming                                                                      
2018-09-18 08:20:56,334  WARN ConfigUtils:523 - SERVER[sandbox-hdp.hortonworks.com] Using a deprecated configuration property [oozie.service.AuthorizationService.security.enabled], should use [oozie.service.Auth
orizationService.authorization.enabled].  Please delete the deprecated property in order for the new property to take effect.                                                                                      
2018-09-18 08:21:07,459  WARN SparkConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Spark Configuration could not be loaded for *: /usr/hdp/current/oozie-server/conf/spark-conf does not exist

Любая помощь? введите описание изображения здесь

...