PySpark Dataframe show метод не работает должным образом

Question

Я пытаюсь прочитать данные из таблицы Hive в кадр данных. При вызове метода dataframe.show возникает исключение NullPointerException.

Кто-нибудь сталкивался с такой же проблемой.

df  = hive.executeQuery("select * from test.employee")

#This line displays the numbers of records as 2
df.count()

#when i try to show the data its throwing null pointer exception.
df.show()

Журналы ошибок:

df.count () 2 df = hive.executeQuery ("select * from sbux_test.employee") Traceback (последний вызов был последним): файл "/usr/hdp/current/spark2-client/python/pyspark/context.py", строка 261 в обработчике сигналов поднять KeyboardInterrupt () KeyboardInterrupt df.show () 19/06/28 08:17:58 ОШИБКА LlapBaseInputFormat: закрытие соединения из-за ошибки shadehive.org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.NullPointerException на shadehive.org.apache.hive.jdbc.Utils.verifySuccess (Utils.java:300) на shadehive.org.apache.hive.jdbc.Utils.verifySuccessWithInfo (Utils.java:286) на shadehive.org.apache.hive.jdbc.HiveQueryResultSet.next (HiveQueryResultSet.java:379) в org.apache.hadoop.hive.llap.LlapBaseInputFormat.getSplits (LlapBaseInputFormat.java:280) в com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.getSplitsFactories (HiveWarehouseDataSourceReader.java:176) в com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.createBatchDataReaderFactories (HiveWarehouseDataSourceReader.java:162) в org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanExec.inputRDD $ lzycompute (DataSourceV2ScanExec.scala: 64) в org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanExec.inputRDD (DataSourceV2ScanExec.scala: 60) в org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanExec.inputRDDs (DataSourceV2ScanExec.scala: 79) в org.apache.spark.sql.execution.ProjectExec.inputRDDs (basicPhysicalOperators.scala: 41) в org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute (WholeStageCodegenExec.scala: 605) в org.apache.spark.sql.execution.SparkPlan $$ anonfun $ execute $ 1.apply (SparkPlan.scala: 131) в org.apache.spark.sql.execution.SparkPlan $$ anonfun $ execute $ 1.apply (SparkPlan.scala: 127) в org.apache.spark.sql.execution.SparkPlan $$ anonfun $ executeQuery $ 1.apply (SparkPlan.scala: 155) в org.apache.spark.rdd.RDDOperationScope $ .withScope (RDDOperationScope.scala: 151) в org.apache.spark.sql.execution.SparkPlan.executeQuery (SparkPlan.scala: 152) в org.apache.spark.sql.execution.SparkPlan.execute (SparkPlan.scala: 127) в org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd (SparkPlan.scala: 247) в org.apache.spark.sql.execution.SparkPlan.executeTake (SparkPlan.scala: 337) в org.apache.spark.sql.execution.CollectLimitExec.executeCollect (limit.scala: 38) в org.apache.spark.sql.Dataset.org $ apache $ spark $ sql $ Набор данных $$ collectFromPlan (Dataset.scala: 3278) в org.apache.spark.sql.Dataset $$ anonfun $ head $ 1.apply (Dataset.scala: 2489) в org.apache.spark.sql.Dataset $$ anonfun $ head $ 1.apply (Dataset.scala: 2489) в org.apache.spark.sql.Dataset $$ anonfun $ 52.apply (Dataset.scala: 3259) в org.apache.spark.sql.execution.SQLExecution $ .withNewExecutionId (SQLExecution.scala: 77) в org.apache.spark.sql.Dataset.withAction (Dataset.scala: 3258) в org.apache.spark.sql.Dataset.head (Dataset.scala: 2489) в org.apache.spark.sql.Dataset.take (Dataset.scala: 2703) в org.apache.spark.sql.Dataset.showString (Dataset.scala: 254) at sun.reflect.NativeMethodAccessorImpl.invoke0 (собственный метод)на солнцеmirror.MethodInvoker.invoke (MethodInvoker.java:244) в py4j.reflection.ReflectionEngine.invoke (ReflectionEngine.java:357) в py4j.Gateway.invoke (Gateway.java:282) в py4j.commands.AbstractCommand.inkeke.java: 132) на py4j.commands.CallCommand.execute (CallCommand.java:79) на py4j.GatewayConnection.run (GatewayConnection.java:238) на java.lang.Thread.run (Thread.java:748), вызванный: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.NullPointerException at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet (SQLOperation.java:478) в org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet (OperationManager.java:328) в org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults (HiveSessionImpl.java:952) в org.apache.hive.service.cli.CLIService.fetchResults (CLIService.java:564) в org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults (ThriftCLIService.java:792) в org.apache.hive.service.rpc.thrift.TCLIService $ Процессор $ FetchResults.getResult (TCLIService.javag: 1837).hive.service.rpc.thrift.TCLIService $ Processor $ FetchResults.getResult (TCLIService.java:1822) в org.apache.thrift.ProcessFunction.process (ProcessFunction.java:39) в org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) в org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge $ Server $ TUGIAssumingProcessor.process (HadoopThriftAuthBridge.java:647) в org.apache.thrift.server.TreadThoServer.java: 286) в java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149) в java.util.concurrent.ThreadPoolExecutor $ Worker.run (ThreadPoolExecutor.java:624) ... еще 1 причина: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.NullPointerException в org.apache.hadoop.hive.ql.exec.FetchTask.fetch (FetchTask.java:162) в org.apache.hadoop.hive.ql.Driver.getResults (Driver.java:2738) в org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults (ReExecDriver.java:229) в org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet (SQLOperation.java:473) ... еще 13 причин. org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.hive.ql.exec.UDTFOperator.process (UDTFOperator.java:116) в org.apache.hadoop.hive.ql.exec.Operator.baseForward (Operator.java:994) в org.apache.hadoop.hive.ql.exec.Operator.forward (Operator.java:940) в org.apache.hadoop.hive.ql.exec.Operator.forward (Operator.java:927) в org.apache.hadoop.hive.ql.exec.SelectOperator.process (SelectOperator.java:95) в org.apache.hadoop.hive.ql.exec.Operator.baseForward (Operator.java:994) в org.apache.hadoop.hive.ql.exec.Operator.forward (Operator.java:940) в org.apache.hadoop.hive.ql.exec.TableScanOperator.process (TableScanOperator.java:125)в org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow (FetchOperator.java:519) в org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow (FetchOperator.java:511) в org.apache.hadoop.hive.ql.exec.FetchTask.fetch (FetchTask.java:146) ... еще 16 причин: java.io.IOException: java.lang.NullPointerException в org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.getSplits (GenericUDTFGetSplits.java:498) в org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process (GenericUDTFGetSplits.java:210) ... еще 27 причин: java.lang.NullPointerException: null в org.apache.hadoop.hive.llap.LlapUtil.generateClusterName (LlapUtil.java:117) в org.apache.hadoop.hive.llap.coordinator.LlapCoordinator.getLlapSigner (LlapCoordinator.java:103) в org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.getSplits (GenericUDTFGetSplits.java:441) ... еще 28 19.0628 08:17:58 ОШИБКА HiveWarehouseDataSourceReader: невозможно отправить запрос в HS2 Traceback (последний вызов был последним): Файл "", строка 1, в файле "/usr/hdp/current/spark2-client/python/pyspark/sql/dataframe.py", строка 350, в шоу print (self._jdf.showString (n, 20, вертикальный)) Файл "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", строка 1257, в вызов Файл "/usr/hdp/current/spark2-client/python/pyspark/sql/utils.py", строка 63, в деко вернуть f (* a, ** kw) файл "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", строка 328 в get_return_value py4j.protocol.Py4JJavaError: ошибка произошло во время вызова o216.showString. : java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: shadehive.org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.NullPointerException в com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.createBatchDataReaderFactories (HiveWarehouseDataSourceReader.java:166) в org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanExec.inputRDD $ lzycompute (DataSourceV2ScanExec.scala: 64) в org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanExec.inputRDD (DataSourceV2ScanExec.scala: 60) в org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanExec.inputRDDs (DataSourceV2ScanExec.scala: 79) в org.apache.spark.sql.execution.ProjectExec.inputRDDs (basicPhysicalOperators.scala: 41) в org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute (WholeStageCodegenExec.scala: 605) в org.apache.spark.sql.execution.SparkPlan $$ anonfun $ execute $ 1.apply (SparkPlan.scala: 131) в org.apache.spark.sql.execution.SparkPlan $$ anonfun $ execute $ 1.apply (SparkPlan.scala: 127) в org.apache.spark.sql.execution.SparkPlan $$ anonfun $ executeQuery $ 1.apply (SparkPlan.scala: 155) в org.apache.spark.rdd.RDDOperationScope $ .withScope (RDDOperationScope.scala: 151) в org.apache.spark.sql.execution.SparkPlan.executeQuery (SparkPlan.scala: 152) в org.apache.spark.sql.execution.SparkPlan.execute (SparkPlan.scala: 127) в org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd (SparkPlan.scala: 247) в org.apache.spark.sql.execution.SparkPlan.executeTake (SparkPlan.scala: 337) в org.apache.spark.sql.execution.CollectLimitExec.executeCollect (limit.scala: 38) в org.apache.spark.sql.Dataset.org $ apache $ spark $ sql $ Набор данных $$ collectFromPlan (Dataset.scala: 3278) в org.apache.spark.sql.Dataset $$ anonfun $ head $ 1.apply (Dataset.scala: 2489)в org.apache.spark.sql.Dataset $$ anonfun $ head $ 1.apply (Dataset.scala: 2489) в org.apache.spark.sql.Dataset $$ anonfun $ 52.apply (Dataset.scala: 3259) в org.apache.spark.sql.execution.SQLExecution $ .withNewExecutionId (SQLExecution.scala: 77) в org.apache.spark.sql.Dataset.withAction (Dataset.scala: 3258) в org.apache.spark.sql.Dataset.голова (Dataset.scala: 2489) в org.apache.spark.sql.Dataset.take (Dataset.scala: 2703) в org.apache.spark.sql.Dataset.showString (Dataset.scala: 254) в sun.reflect.NativeMethodAccessorImpl.invoke0 (родной метод) в sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) в sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegavaMetho.rej.rej.rej.ref.jj.java: 498) в py4j.reflection.MethodInvoker.invoke (MethodInvoker.java:244) в py4j.reflection.ReflectionEngine.invoke (ReflectionEngine.java:357) в py4j.Gateway.invoke (Gateway.java:282) в py4.commands.AbstractCommand.invokeMethod (AbstractCommand.java:132) на py4j.commands.CallCommand.execute (CallCommand.java:79) на py4j.GatewayConnection.run (GatewayConnection.java:238) на java.lang.Thread.run (Thread.java:748). Причина: java.lang.RuntimeException: java.io.IOException: shadehive.org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:java.lang.NullPointerException в com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.getSplitsFactories (HiveWarehouseDataSourceReader.java:182) в com.hortonworks.spark.sql.hive.llap.ReataSourceReader) ... еще 34 Причина: java.io.IOException: shadehive.org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.NullPointerException в org.apache.hadoop.hive.llap.LlapBaseInputFormat.getSplits (LlapBaseInputFormat.java:298) в com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.getSplitsFactories (HiveWarehouseDataSourceReader.java:176) ... еще 35 Причин:apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.NullPointerException в shadehive.org.apache.hive.jdbc.Utils.verifySuccess (Utils.java:300) в shadehive.org.apache.hive.(LlapBaseInputFormat.java:280) ... еще 36 причин: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.NullPointerException в org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet (SQLOperation.java:478) в org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet (OperationManager.java:328) в org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults (HiveSessionImpl.java:952) в org.apache.hive.service.cli.CLIService.fetchResults (CLIService.java:564) в org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults (ThriftCLIService.java:792)в org.apache.hive.service.rpc.thrift.TCLIService $ Processor $ FetchResults.getResult (TCLIService.java:1837) в org.apache.hive.service.rpc.thrift.TCLIService $ Processor $ FetchResults.getResult (TCLIService.java:1822) в org.apache.thrift.ProcessFunction.process (ProcessFunction.java:39) в org.apache.thrift.TBaseProcessor.process (TBaseProcessor.java:39) в org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge $ Server $ TUGIAssumingProcessor.process (HadoopThriftAuthBridge.java:647) в org.apache.thrift.server.TThreadPoolServer $ WorkerProcess.run (TThreadPoolServer.java:286) в java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149) в java.util.concurrent.ThreadPoolExecutor $ Worker.run (ThreadPoolExecutor.java:624) ... еще 1 причина: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.NullPointerException в org.apache.hadoop.hive.ql.exec.FetchTask.fetch (FetchTask.java:162) в org.apache.hadoop.hive.ql.Driver.getResults (Driver.java:2738) в org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults (ReExecDriver.java:229) в org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet (SQLOperation.java:473) ... еще 13 Причина: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.NullPointerException в org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process (GenericUDTFGetSplits.java:225) в org.apache.hadoop.hive.ql.exec.UDTFOperator.process (UDTFOperator.java:116) в org.apache.hadoop.hive.ql.exec.Operator.baseForward (Operator.java:994) на org.apache.hadoop.hive.ql.exec.Operator.forward (Operator.java:940) в org.apache.hadoop.hive.ql.exec.Operator.forward (Operator.java:927) в org.apache.hadoop.hive.ql.exec.SelectOperator.process (SelectOperator.java:95) в org.apache.hadoop.hive.ql.exec.Operator.baseForward (Operator.java:994) на org.apache.hadoop.hive.ql.exec.Operator.forward (Operator.java:940) в org.apache.hadoop.hive.ql.exec.TableScanOperator.process (TableScanOperator.java:125) в org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow (FetchOperator.java:519) в org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow (FetchOperator.java:511) в org.apache.hadoop.hive.ql.exec.FetchTask.fetch (FetchTask.java:146) ... еще 16 причин: java.io.IOException: java.lang.NullPointerException в org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.getSplits (GenericUDTFGetSplits.java:498) в org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process (GenericUDTFGetSplits.java:210) ... еще 27 причин: java.lang.NullPointerException: null в org.apache.hadoop.hive.llap.LlapUtil.generateClusterName (LlapUtil.java:117) в org.apache.hadoop.hive.llap.coordinator.LlapCoordinator.getLlapSigner (LlapCoordinator.java:103) в org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.getSplits (GenericUDTFGetSplits.java:441) ... еще 28