Я пытаюсь импортировать данные из таблицы базы данных HyperSQL в Hadoop DFS, используя Sqoop версии 1.4.6.2.6.5.0-292.Я попытаюсь обобщить шаги, которые я предпринял:
Сначала я скачал и распаковал HSQL версии 1.8.0.10 (это та же версия, что и в Sqoop).Я добавил файл server.properties и настроил папку базы данных следующим образом: server.database.0=file:test/test
Служба базы данных была запущена следующим образом:
nohup java -cp /data/programs/ETU/hsqldb-1.8.0/hsqldb/lib/hsqldb.jar org.hsqldb.Server &
После создания таблицы с именем RT_SALES и заполнения ее данными, я смог установить успешное соединение из sqoop, используя
sqoop list-tables --connect "jdbc:hsqldb:hsql://localhost" --connection-manager "org.apache.sqoop.manager.HsqldbManager" --username "SA" В результате я получил желаемую таблицу RT_SALES.Однако при попытке импортировать данные в HDFS с использованием
sqoop import --num-mappers 1 --connect "jdbc:hsqldb:hsql://localhost" --connection-manager "org.apache.sqoop.manager.HsqldbManager" --username "SA" --table "RT_SALES" --target-dir "/some-folder"
я получаю следующую ошибку:
19/05/08 20:16:13 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.6.5.0-292
19/05/08 20:16:13 INFO manager.SqlManager: Using default fetchSize of 1000
19/05/08 20:16:13 INFO tool.CodeGenTool: Beginning code generation
19/05/08 20:16:14 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM "RT_SALES" AS t WHERE 1=0
19/05/08 20:16:14 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM "RT_SALES" AS t WHERE 1=0
19/05/08 20:16:14 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.6.5.0-292/hadoop-mapreduce
Note: /tmp/sqoop-user1/compile/50b3c7f6e0257e6f477a23139bedcb39/RT_SALES.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
19/05/08 20:16:16 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-user1/compile/50b3c7f6e0257e6f477a23139bedcb39/RT_SALES.jar
19/05/08 20:16:16 INFO mapreduce.ImportJobBase: Beginning import of RT_SALES
19/05/08 20:16:17 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM "RT_SALES" AS t WHERE 1=0
19/05/08 20:16:18 INFO client.AHSProxy: Connecting to Application History server at server1/100.00.000.00:10200
19/05/08 20:16:18 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 70501 for user1 on ha-hdfs:server1
19/05/08 20:16:18 INFO security.TokenCache: Got dt for hdfs://server1; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:server1, Ident: (HDFS_DELEGATION_TOKEN token 70501 for user1)
19/05/08 20:16:18 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
19/05/08 20:16:18 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]
19/05/08 20:16:21 INFO db.DBInputFormat: Using read commited transaction isolation
19/05/08 20:16:21 INFO mapreduce.JobSubmitter: number of splits:1
19/05/08 20:16:21 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1557311883605_0041
19/05/08 20:16:21 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:server1, Ident: (HDFS_DELEGATION_TOKEN token 70501 for user1)
19/05/08 20:16:22 INFO impl.TimelineClientImpl: Timeline service address: http://server1:8188/ws/v1/timeline/
19/05/08 20:16:23 INFO impl.YarnClientImpl: Submitted application application_1557311883605_0041
19/05/08 20:16:23 INFO mapreduce.Job: The url to track the job: http://server1:8088/proxy/application_1557311883605_0041/
19/05/08 20:16:23 INFO mapreduce.Job: Running job: job_1557311883605_0041
19/05/08 20:16:31 INFO mapreduce.Job: Job job_1557311883605_0041 running in uber mode : false
19/05/08 20:16:31 INFO mapreduce.Job: map 0% reduce 0%
19/05/08 20:16:37 INFO mapreduce.Job: Task Id : attempt_1557311883605_0041_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: socket creation error
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
Caused by: java.lang.RuntimeException: java.sql.SQLException: socket creation error
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
... 9 more
Caused by: java.sql.SQLException: socket creation error
at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
at org.hsqldb.jdbc.jdbcConnection.<init>(Unknown Source)
at org.hsqldb.jdbcDriver.getConnection(Unknown Source)
at org.hsqldb.jdbcDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more
Судя по другим сообщениям, сообщение говорит о том, что HSQLDB не запущен.Тем не менее, я могу подключиться с помощью встроенного клиента HSQL, а также, как указано выше, sqoop может по крайней мере запрашивать таблицы - поэтому служба должна быть запущена.
Я уже пробовал (без успеха):
- определение другого порта (9002 вместо 9001) для hsqldb
- создание пароля для пользователя SA
- установка dbname ('test' вместо '')
Обновление: После того, как для свойства сервера установлено значение false, я получил следующий журнал:
[Server@49476842]: [Thread[main,5,main]]: checkRunning(false) entered
[Server@49476842]: [Thread[main,5,main]]: checkRunning(false) exited
[Server@49476842]: Startup sequence initiated from main() method
[Server@49476842]: Loaded properties from [/data/programs/ETU/hsqldb-1.8.0/hsqldb/server.properties]
[Server@49476842]: [Thread[main,5,main]]: start() entered
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: run() entered
[Server@49476842]: Initiating startup sequence...
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.tls=false
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.port=9002
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.trace=true
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.database.0=file:test/test
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.restart_on_shutdown=false
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.no_system_exit=false
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.silent=false
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.default_page=index.html
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.address=0.0.0.0
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.dbname.0=test
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.root=.
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: openServerSocket() entered
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: Got server socket: ServerSocket[addr=0.0.0.0/0.0.0.0,localport=9002]
[Server@49476842]: Server socket opened successfully in 5 ms.
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: openServerSocket() exiting
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: openDatabases() entered
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: Opening database: [file:test/test]
[Server@49476842]: Database [index=0, id=0, db=file:test/test, alias=test] opened sucessfully in 118 ms.
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: openDatabases() exiting
[Server@49476842]: Startup sequence completed in 124 ms.
[Server@49476842]: 2019-05-10 12:59:46.675 HSQLDB server 1.8.0 is online
[Server@49476842]: To close normally, connect and execute SHUTDOWN SQL
[Server@49476842]: From command line, use [Ctrl]+[C] to abort abruptly
[Server@49476842]: [Thread[main,5,main]]: start() exiting
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: handleConnection(Socket[addr=/127.0.0.1,port=38030,localport=9002]) entered
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: handleConnection() exited
[Server@49476842]: [Thread[HSQLDB Connection @5e60432f,5,HSQLDB Connections @49476842]]: 0:trying to connect user SA
[Server@49476842]: 0:HSQLCLI:SETSESSIONATTR:AUTOCOMMIT null CONNECTION_READONLY null
[Server@49476842]: 0:HSQLCLI:SETSESSIONATTR:AUTOCOMMIT false CONNECTION_READONLY null
[Server@49476842]: 0:SQLCLI:SQLPREPARE SELECT t.* FROM "RT_SALES" AS t WHERE 1=0
[Server@49476842]: 0:SQLCLI:SQLEXECUTE:1
[Server@49476842]: 0:SQLCLI:SQLENDTRAN:COMMIT
[Server@49476842]: 0:SQLCLI:SQLFREESTMT:1
[Server@49476842]: 0:SQLCLI:SQLPREPARE SELECT t.* FROM "RT_SALES" AS t WHERE 1=0
[Server@49476842]: 0:SQLCLI:SQLEXECUTE:2
[Server@49476842]: 0:SQLCLI:SQLENDTRAN:COMMIT
[Server@49476842]: 0:SQLCLI:SQLFREESTMT:2
[Server@49476842]: 0:SELECT * FROM INFORMATION_SCHEMA.SYSTEM_PRIMARYKEYS WHERE 1=1 AND TABLE_NAME = 'RT_SALES'
[Server@49476842]: 0:SQLCLI:SQLENDTRAN:COMMIT
[Server@49476842]: 0:SELECT * FROM INFORMATION_SCHEMA.SYSTEM_PRIMARYKEYS WHERE 1=1 AND TABLE_NAME = 'RT_SALES'
[Server@49476842]: 0:SQLCLI:SQLENDTRAN:COMMIT
[Server@49476842]: 0:SQLCLI:SQLPREPARE SELECT t.* FROM "RT_SALES" AS t WHERE 1=0
[Server@49476842]: 0:SQLCLI:SQLEXECUTE:3
[Server@49476842]: 0:SQLCLI:SQLENDTRAN:COMMIT
[Server@49476842]: 0:SQLCLI:SQLFREESTMT:3
[Server@49476842]: 0:SQLCLI:SQLDISCONNECT
[Server@49476842]: [Thread[HSQLDB Connection @5e60432f,5,HSQLDB Connections @49476842]]: 0:disconnected SA
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: handleConnection(Socket[addr=/127.0.0.1,port=38190,localport=9002]) entered
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: handleConnection() exited
[Server@49476842]: [Thread[HSQLDB Connection @7450cadd,5,HSQLDB Connections @49476842]]: 1:trying to connect user SA
[Server@49476842]: 1:HSQLCLI:SETSESSIONATTR:AUTOCOMMIT false CONNECTION_READONLY null
[Server@49476842]: 1:call "org.hsqldb.Library.getDatabaseProductName"()
[Server@49476842]: 1:HSQLCLI:SETSESSIONATTR:AUTOCOMMIT null CONNECTION_READONLY null
[Server@49476842]: 1:SQLCLI:SQLDISCONNECT
[Server@49476842]: [Thread[HSQLDB Connection @7450cadd,5,HSQLDB Connections @49476842]]: 1:disconnected SA