В настоящее время я работаю над проектом, в котором мне нужно читать таблицу Куду с помощью spark. Когда я пытаюсь прочитать таблицу Куду с помощью spark-shell, я могу читать. Но когда я создал jar приложения и отправил то же самое, используя параметр spark2-submit, это выдает ошибку ниже. Наш кластер керберизован. Не могли бы вы помочь мне в следующем вопросе.
Версия Spark: 2.4
Версия Kudu: 1.6
Scala Версия: 2.11
Команда:
spark2-submit --master yarn --deploy-mode cluster \
--class org.xxx.xxx.xxx \
--jars "/home/xxx/geospark-1.2.0.jar,/home/xxx/geospark/geospark-sql_2.3-1.2.0.jar,/home/xxx/geospark/geospark-viz_2.3-1.2.0.jar,/home/xxx/geospark/kudu-spark2_2.11-1.9.0-cdh6.2.0.jar" \
/home/xxx/xxx-0.0.1-SNAPSHOT.jar
--principal user@XXX.LOCAL \
--keytab /home/xxx/user.keytab
Вывод:
20/04/02 17:06:56 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
20/04/02 17:06:56 INFO yarn.ApplicationMaster: Waiting for spark context initialization...
20/04/02 17:06:59 ERROR yarn.ApplicationMaster: User class threw exception: java.security.PrivilegedActionException: org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (xxx.xxx.xxx.xxx:7051). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available]
java.security.PrivilegedActionException: org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (xxx.xxx.xxx.xxx:7051). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available]
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.kudu.spark.kudu.KuduContext.<init>(KuduContext.scala:145)
at org.apache.kudu.spark.kudu.KuduRelation.<init>(DefaultSource.scala:256)
at org.apache.kudu.spark.kudu.DefaultSource.createRelation(DefaultSource.scala:109)
at org.apache.kudu.spark.kudu.DefaultSource.createRelation(DefaultSource.scala:87)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
at org.apache.kudu.spark.kudu.package$KuduDataFrameReader.kudu(package.scala:32)
at org.apache.GFence.GeoFence$.main(GeoFence.scala:47)
at org.apache.GFence.GeoFence.main(GeoFence.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)```