Как устранить конфликтующие зависимости коннекторов Twitter и PubSub - PullRequest
Новая
       84

Как устранить конфликтующие зависимости коннекторов Twitter и PubSub

0 голосов
/ 25 сентября 2019

Я пытаюсь построить конвейер Flink, который читает из API Twitter в реальном времени и пишет в Google PubSub.

Вот мой проект sbt: 

val flinkVersion = "1.9.0"

lazy val twitter = project
  .settings(
    name := "twitter",
    assembly / mainClass := Some("twitter_realtime_events.TwitterApp"),
    libraryDependencies ++= Seq(
      "org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
      "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided",
      "org.apache.flink" %% "flink-connector-twitter" % flinkVersion,
      "org.apache.flink" %% "flink-connector-gcp-pubsub" % flinkVersion,
    ),
  )

Когда я бегу assembly Я получаю ошибки: 

[error] 718 errors were encountered during merge
[error] stack trace is suppressed; run last assembly for the full output
[error] (assembly) deduplicate: different file contents found in the following:
[error] /Users/krinart/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/io/grpc/grpc-netty-shaded/1.17.1/grpc-netty-shaded-1.17.1.jar:META-INF/io.netty.versions.properties
[error] /Users/krinart/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/flink/flink-shaded-netty/4.1.32.Final-7.0/flink-shaded-netty-4.1.32.Final-7.0.jar:META-INF/io.netty.versions.properties
[error] deduplicate: different file contents found in the following:
[error] /Users/krinart/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/4.5.5/httpclient-4.5.5.jar:mozilla/public-suffix-list.txt
[error] /Users/krinart/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/flink/flink-connector-twitter_2.12/1.9.0/flink-connector-twitter_2.12-1.9.0.jar:mozilla/public-suffix-list.txt
[error] deduplicate: different file contents found in the following:
[error] /Users/krinart/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/httpcomponents/httpcore/4.4.9/httpcore-4.4.9.jar:org/apache/http/ConnectionClosedException.class
[error] /Users/krinart/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/flink/flink-connector-twitter_2.12/1.9.0/flink-connector-twitter_2.12-1.9.0.jar:org/apache/http/ConnectionClosedException.class
[error] deduplicate: different file contents found in the following:
[error] /Users/krinart/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/httpcomponents/httpcore/4.4.9/httpcore-4.4.9.jar:org/apache/http/ConnectionReuseStrategy.class
[error] /Users/krinart/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/flink/flink-connector-twitter_2.12/1.9.0/flink-connector-twitter_2.12-1.9.0.jar:org/apache/http/ConnectionReuseStrategy.class
[error] deduplicate: different file contents found in the following:
[error] /Users/krinart/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/httpcomponents/httpcore/4.4.9/httpcore-4.4.9.jar:org/apache/http/Consts.class
[error] /Users/krinart/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/flink/flink-connector-twitter_2.12/1.9.0/flink-connector-twitter_2.12-1.9.0.jar:org/apache/http/Consts.class
...

Понятно, что и flink-connector-twitter, и flink-connector-gcp-pubsub имеют одинаковую зависимость - org.apache.httpcomponents, однако flink-connector-twitter по какой-то причине, кажется, упаковал его в свой толстый сосуд.что вызывает ошибки выше.

Что я могу сделать, чтобы разрешить этот конфликт?

...