Команда sqoop действий оболочки Oozie Sqoop не найдена - PullRequest
Новая
       40

Команда sqoop действий оболочки Oozie Sqoop не найдена

0 голосов
/ 16 октября 2019

Я пытаюсь выполнить действие оболочки sqoop. Сценарий оболочки прекрасно работает на локальном узле, но при запуске в oozie отображается ошибка, что команда sqoop не найдена. 

NM_AUX_SERVICE_spark_shuffle=:
LOCAL_USER_DIRS=/data/disk1/hadoop/yarn/local/usercache/206638483/,/data/disk2/hadoop/yarn/local/usercache/206638483/,/data/disk3/hadoop/yarn/local/usercache/206638483/,/data/disk4/hadoop/yarn/local/usercache/206638483/,/data/disk5/hadoop/yarn/local/usercache/206638483/,/data/disk6/hadoop/yarn/local/usercache/206638483/,/data/disk7/hadoop/yarn/local/usercache/206638483/,/data/disk8/hadoop/yarn/local/usercache/206638483/,/data/disk9/hadoop/yarn/local/usercache/206638483/,/data/disk10/hadoop/yarn/local/usercache/206638483/,/data/disk11/hadoop/yarn/local/usercache/206638483/,/data/disk12/hadoop/yarn/local/usercache/206638483/,/data/disk13/hadoop/yarn/local/usercache/206638483/,/data/disk14/hadoop/yarn/local/usercache/206638483/,/data/disk15/hadoop/yarn/local/usercache/206638483/,/data/disk16/hadoop/yarn/local/usercache/206638483/,/data/disk17/hadoop/yarn/local/usercache/206638483/,/data/disk18/hadoop/yarn/local/usercache/206638483/,/data/disk19/hadoop/yarn/local/usercache/206638483/,/data/disk20/hadoop/yarn/local/usercache/206638483/,/data/disk21/hadoop/yarn/local/usercache/206638483/,/data/disk22/hadoop/yarn/local/usercache/206638483/,/data/disk23/hadoop/yarn/local/usercache/206638483/,/data/disk24/hadoop/yarn/local/usercache/206638483/:
HADOOP_HOME=/usr/hdp/2.6.4.125-1/hadoop:
OOZIE_ACTION_CONF_XML=/data/disk1/hadoop/yarn/local/usercache/206638483/appcache/application_1559931840942_7387/container_e45_1559931840942_7387_01_000002/action.xml:
SHLVL=1:
HOME=/home/:
NM_AUX_SERVICE_spark2_shuffle=:
CONTAINER_ID=container_e45_1559931840942_7387_01_000002:
MALLOC_ARENA_MAX=4:
=================================================================

>>> Invoking Shell command line now >>

Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Heart beat
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Heart beat
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Heart beat
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput 
Exit code of the Shell command 0
<<< Invocation of Shell command completed <<<


<<< Invocation of Main class completed <<<

Successfully reset security manager from org.apache.oozie.action.hadoop.LauncherSecurityManager@5a4bef8 to null

Oozie Launcher ends

Workflow.xml: 

   ?xml version="1.0" encoding="UTF-8" standalone="no"?><workflow-app 
   xmlns="uri:oozie:workflow:0.5" name="Edm_Sqoop_full">
    <start to="shell_1"/>
    <action name="shell_1">
        <shell xmlns="uri:oozie:shell-action:0.3">
            <job-tracker>${resourceManager}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>oozie.use.system.libpath</name>
                    <value>true</value>
                </property>
            </configuration>
            <exec>/user/sparkcronsvc/edm/sqoop_full.sh</exec>
            <argument>uso12smktdb01</argument>
            <argument>unica</argument>
            <file>/user/sparkcronsvc/edm/sqoop_full.sh</file>
            <file>/user/sparkcronsvc/edm/fulllist.txt</file>
        </shell>
        <ok to="end"/>
        <error to="kill"/>
    </action>
    <kill name="kill">
        <message>${wf:errorMessage(wf:lastErrorNode())}</message>
    </kill>
    <end name="end"/>
    </workflow-app>

Пожалуйста, предложите. Сценарий оболочки вызывает импорт sqoop с зашифрованными рывками паролей и именами таблиц через список таблиц. 

Script is :

#! / Bin / sh dmserver = $ 1 hdp_dir = $ 2

при чтении -r строки сделать tbl = echo $line | cut -d "|" -f1 dmdb = echo $line | cut -d "|" -f2 sqoop import -Dmapreduce.job. user.classpath.first = true - Dhadoop.security.credential.provider.path = jceks: //hdfs/user/root/mssql.datamarthdpservice.pass.jceks \ --connect 

  'jdbc:jtds:sqlserver://'${server}'.use.ucdp.net/'${dmdb}';
   integratedSecurity=true;domain=use.ucdp.net;database='${dmdb}';' \
                      --username dataid \
                     --password-alias mssql.data.pass \
                     --query "SELECT * from ${dmdb}.dbo.${tbl} WITH(NOLOCK) 
   WHERE \$CONDITIONS" -m 1 \
                    -----------

   done < <(tail -n +2 fulllist.txt | grep -v "#") 
```

1 Ответ

0 голосов
/ 01 ноября 2019

Кажется, что командная строка sqoop не установлена ​​в каждом узле данных, это должно быть сбой. Делаем то же самое с действием sqoop.

...