Блоки данных SFTP - Nullpointer - PullRequest
Новая
       16

Блоки данных SFTP - Nullpointer

1 голос
/ 22 марта 2019

Я использую библиотеку, расположенную по адресу: https://github.com/springml/spark-sftp

Используя команду (конечно, используя мои учетные данные): 

df.write.
      format("com.springml.spark.sftp").
      option("host", "SFTP_HOST").
      option("username", "SFTP_USER").
      option("password", "****").
      option("fileType", "csv").
      option("delimiter", ";").
      option("codec", "bzip2").
      save("/ftp/files/sample.csv")

Однако я получаю только ошибку: 

java.lang.NullPointerException
    at scala.collection.mutable.ArrayOps$ofRef$.newBuilder$extension(ArrayOps.scala:190)
    at scala.collection.mutable.ArrayOps$ofRef.newBuilder(ArrayOps.scala:186)
    at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:246)
    at scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
    at scala.collection.mutable.ArrayOps$ofRef.filter(ArrayOps.scala:186)
    at com.springml.spark.sftp.DefaultSource.copiedFile(DefaultSource.scala:276)
    at com.springml.spark.sftp.DefaultSource.writeToTemp(DefaultSource.scala:264)
    at com.springml.spark.sftp.DefaultSource.createRelation(DefaultSource.scala:130)
    at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:72)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:70)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:88)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:143)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$5.apply(SparkPlan.scala:183)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:180)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:131)
    at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:114)
    at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:114)
    at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:690)
    at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:690)

Кто-нибудь успешно запускал это в блоках данных?

...