ошибка при импорте проекта sbt для apache phoenix

Question

Я новичок, я пытаюсь использовать sbt для импорта библиотек phoenix для чтения таблиц hbase в spark, но мой build.sbt постоянно выдает мне ошибки.

Ошибка при импорте проекта sbt:

[error] stack trace is suppressed; run 'last update' for the full output
[error] stack trace is suppressed; run 'last ssExtractDependencies' for the full output
[error] (update) sbt.librarymanagement.ResolveException: Error downloading org.apache.hbase:hbase-common:${cdh.hbase.version}
[error]   Not found
[error]   Not found
[error]   not found: /Users/johnny/.ivy2/local/org.apache.hbase/hbase-common/${cdh.hbase.version}/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/org/apache/hbase/hbase-common/${cdh.hbase.version}/hbase-common-${cdh.hbase.version}.pom
[error] Error downloading org.apache.hbase:hbase-hadoop-compat:${cdh.hbase.version}
[error]   Not found
[error]   Not found
[error]   not found: /Users/johnny/.ivy2/local/org.apache.hbase/hbase-hadoop-compat/${cdh.hbase.version}/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/org/apache/hbase/hbase-hadoop-compat/${cdh.hbase.version}/hbase-hadoop-compat-${cdh.hbase.version}.pom

[error]   not found: https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/${cdh.hadoop.version}/hadoop-common-${cdh.hadoop.version}.pom
[error] Error downloading org.apache.hbase:hbase-hadoop2-compat:${cdh.hbase.version}
[error]   Not found
[error]   Not found
[error]   not found: /Users/johnny/.ivy2/local/org.apache.hbase/hbase-hadoop2-compat/${cdh.hbase.version}/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/org/apache/hbase/hbase-hadoop2-compat/${cdh.hbase.version}/hbase-hadoop2-compat-${cdh.hbase.version}.pom
[error] Error downloading org.apache.hbase:hbase-annotations:${cdh.hbase.version}
[error]   Not found
[error]   Not found
[error]   not found: /Users/johnny/.ivy2/local/org.apache.hbase/hbase-annotations/${cdh.hbase.version}/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/org/apache/hbase/hbase-annotations/${cdh.hbase.version}/hbase-annotations-${cdh.hbase.version}.pom
[error] Error downloading org.apache.hbase:hbase-protocol:${cdh.hbase.version}
[error]   Not found
[error]   Not found
[error]   not found: /Users/johnny/.ivy2/local/org.apache.hbase/hbase-protocol/${cdh.hbase.version}/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/org/apache/hbase/hbase-protocol/${cdh.hbase.version}/hbase-protocol-${cdh.hbase.version}.pom
[error] Error downloading org.apache.hbase:hbase-client:${cdh.hbase.version}
[error]   Not found
[error]   Not found
[error]   not found: /Users/johnny/.ivy2/local/org.apache.hbase/hbase-client/${cdh.hbase.version}/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/org/apache/hbase/hbase-client/${cdh.hbase.version}/hbase-client-${cdh.hbase.version}.pom
[error] Error downloading org.apache.hbase:hbase-server:${cdh.hbase.version}
[error]   Not found
[error]   Not found
[error]   not found: /Users/johnny/.ivy2/local/org.apache.hbase/hbase-server/${cdh.hbase.version}/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/org/apache/hbase/hbase-server/${cdh.hbase.version}/hbase-server-${cdh.hbase.version}.pom
[error] Error downloading com.cloudera.cdh:cdh-root:5.11.2
[error]   Not found
[error]   Not found
[error]   not found: /Users/johnny/.ivy2/local/com.cloudera.cdh/cdh-root/5.11.2/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/com/cloudera/cdh/cdh-root/5.11.2/cdh-root-5.11.2.pom
[error] Total time: 3 s, completed Sep 27, 2019, 4:54:09 PM
[info] shutting down sbt server)

Мой build.sbt:

name := "SparkHbase"
version := "0.1"
scalaVersion := "2.11.12"
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.2.0" % "provided"
  ,"org.apache.spark" %% "spark-sql" % "2.2.0"  % "provided"
  ,"org.apache.spark" %% "spark-hive" % "2.2.0" % "provided"
  ,"org.apache.phoenix" % "phoenix-spark" % "4.13.2-cdh5.11.2"
)

Я даже включил это: resolvers += "ClouderaRepo" at "https://repository.cloudera.com/content/repositories/releases"

Но все равно были ошибки.Пожалуйста, что я делаю не так?

Answer 1

Проблема в том, что вы пытаетесь использовать очень старую версию phoenix-spark.Если у вас HBase 1.3, вы можете использовать версию 4.14.3-HBase-1.3, см. build.sbt:

name := "SparkHbase"
version := "0.1"
scalaVersion := "2.11.12"

resolvers += "Cloudera" at "https://repository.cloudera.com/content/repositories/releases/"
resolvers += "Cloudera_Artifactory" at "https://repository.cloudera.com/artifactory/cloudera-repos/"
resolvers += Resolver.sonatypeRepo("releases")

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.2.0" % "provided"
  ,"org.apache.spark" %% "spark-sql" % "2.2.0"  % "provided"
  ,"org.apache.spark" %% "spark-hive" % "2.2.0" % "provided"
  ,"org.apache.phoenix" % "phoenix-spark" % "4.14.3-HBase-1.3"
)