Работа Oozie с sqoop action зависает - PullRequest
Новая
       28

Работа Oozie с sqoop action зависает

0 голосов
/ 28 октября 2018

В виртуальной машине cloudera.quickstart я изучаю oozie и пытаюсь запустить простой рабочий процесс с действием sqoop.Когда я выполняю работу, кажется, что она зависает, и я не понимаю, что там происходит.Я не замечаю ошибку или URL-адрес отсутствует в уведомлении: 

2018-10-28 03:23:26,481  INFO JPAService:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000003-181027074351697-oozie-oozi-W] ACTION[] No results found
2018-10-28 03:23:26,575  INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000003-181027074351697-oozie-oozi-W] ACTION[0000003-181027074351697-oozie-oozi-W@:start:] Start action [0000003-181027074351697-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-10-28 03:23:26,576  INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000003-181027074351697-oozie-oozi-W] ACTION[0000003-181027074351697-oozie-oozi-W@:start:] [***0000003-181027074351697-oozie-oozi-W@:start:***]Action status=DONE
2018-10-28 03:23:26,576  INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000003-181027074351697-oozie-oozi-W] ACTION[0000003-181027074351697-oozie-oozi-W@:start:] [***0000003-181027074351697-oozie-oozi-W@:start:***]Action updated in DB!
2018-10-28 03:23:26,690  INFO JPAService:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000003-181027074351697-oozie-oozi-W] ACTION[0000003-181027074351697-oozie-oozi-W@:start:] No results found
2018-10-28 03:23:26,740  INFO WorkflowNotificationXCommand:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-181027074351697-oozie-oozi-W] ACTION[0000003-181027074351697-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000003-181027074351697-oozie-oozi-W@:start:
2018-10-28 03:23:26,740  INFO WorkflowNotificationXCommand:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-181027074351697-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000003-181027074351697-oozie-oozi-W
2018-10-28 03:23:26,812  INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000003-181027074351697-oozie-oozi-W] ACTION[0000003-181027074351697-oozie-oozi-W@sqoop-node] Start action [0000003-181027074351697-oozie-oozi-W@sqoop-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]

Определение рабочего процесса: 

<?xml version="1.0" encoding="UTF-8"?>

<workflow-app xmlns="uri:oozie:workflow:0.2" name="sqoop-wf">
    <start to="sqoop-node"/>

    <action name="sqoop-node">
        <sqoop xmlns="uri:oozie:sqoop-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/sqoop"/>
                <mkdir path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data"/>
            </prepare>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <command>import --connect jdbc:mysql://localhost:3306/retail_db --username=retail_dba --password cloudera \
--table departments --as-textfile --target-dir /user/cloudera/departments</command>
        </sqoop>
        <ok to="end"/>
        <error to="fail"/>
    </action>

    <kill name="fail">
        <message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

файл свойств 

nameNode=hdfs://localhost:8020
jobTracker=localhost:8023
queueName=default
examplesRoot=questions_oozie/q59

oozie.use.system.libpath=true

oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/

Добавление: я создал еще один рабочий процесс oozie, на этот раз используя HUE и Pig Script.Таким образом, рабочий процесс был сгенерирован HUE и не должен содержать ошибок.Я предполагаю, что проблема связана с конфигурацией моего кластера? 

2018-10-28 07:01:53,110  INFO JPAService:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[] No results found
2018-10-28 07:01:53,240  INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@:start:] Start action [0000007-181027074351697-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-10-28 07:01:53,244  INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@:start:] [***0000007-181027074351697-oozie-oozi-W@:start:***]Action status=DONE
2018-10-28 07:01:53,244  INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@:start:] [***0000007-181027074351697-oozie-oozi-W@:start:***]Action updated in DB!
2018-10-28 07:01:53,428  INFO JPAService:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@:start:] No results found
2018-10-28 07:01:53,478  INFO WorkflowNotificationXCommand:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000007-181027074351697-oozie-oozi-W@:start:
2018-10-28 07:01:53,479  INFO WorkflowNotificationXCommand:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000007-181027074351697-oozie-oozi-W
2018-10-28 07:01:53,564  INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] Start action [0000007-181027074351697-oozie-oozi-W@pig-ca9a] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-10-28 07:01:55,590  INFO PigActionExecutor:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] checking action, hadoop job ID [job_1540651330075_0018] status [RUNNING]
2018-10-28 07:01:55,600  INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] [***0000007-181027074351697-oozie-oozi-W@pig-ca9a***]Action status=RUNNING
2018-10-28 07:01:55,600  INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] [***0000007-181027074351697-oozie-oozi-W@pig-ca9a***]Action updated in DB!
2018-10-28 07:01:55,666  INFO WorkflowNotificationXCommand:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] No Notification URL is defined. Therefore nothing to notify for job 0000007-181027074351697-oozie-oozi-W@pig-ca9a
2018-10-28 07:05:30,298  INFO CallbackServlet:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] callback for action [0000007-181027074351697-oozie-oozi-W@pig-ca9a]
2018-10-28 07:05:30,696  INFO PigActionExecutor:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] Hadoop Jobs launched : [job_1540651330075_0020,job_1540651330075_0021]
2018-10-28 07:05:30,701  INFO PigActionExecutor:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] action completed, external ID [job_1540651330075_0018]
2018-10-28 07:05:30,725  WARN PigActionExecutor:523 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]
2018-10-28 07:05:30,772  INFO ActionEndXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] ERROR is considered as FAILED for SLA
2018-10-28 07:05:30,834  INFO JPAService:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] No results found
2018-10-28 07:05:30,875  INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@Kill] Start action [0000007-181027074351697-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-10-28 07:05:30,875  INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@Kill] [***0000007-181027074351697-oozie-oozi-W@Kill***]Action status=DONE
2018-10-28 07:05:30,875  INFO ActionStartXCommand:520 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@Kill] [***0000007-181027074351697-oozie-oozi-W@Kill***]Action updated in DB!
2018-10-28 07:05:31,069  INFO WorkflowNotificationXCommand:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@Kill] No Notification URL is defined. Therefore nothing to notify for job 0000007-181027074351697-oozie-oozi-W@Kill
2018-10-28 07:05:31,071  INFO WorkflowNotificationXCommand:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000007-181027074351697-oozie-oozi-W
2018-10-28 07:05:31,072  INFO WorkflowNotificationXCommand:520 - SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000007-181027074351697-oozie-oozi-W] ACTION[0000007-181027074351697-oozie-oozi-W@pig-ca9a] No Notification URL is defined. Therefore nothing to notify for job 0000007-181027074351697-oozie-oozi-W@pig-ca9a
...