Интеграция облачного Firestore в облачный поток данных в Google Cloud - PullRequest
Новая
       19

Интеграция облачного Firestore в облачный поток данных в Google Cloud

0 голосов
/ 09 мая 2019

Я интегрирую в облачный поток данных приложение, которое записывает сообщение json в Cloud Firestore. Проблема в том, что библиотека лучей apache (и зависимости) и библиотека Firestore несовместимы. Ниже я покажу вам выдержку из моего pom, кода потока данных и ошибки компиляции maven:

Поток данных работает исключительно хорошо при чтении из паба / подпрограммы или из облачного хранилища и записи в паб / подпункт Но в тот момент, когда я добавляю зависимость firestore, я получаю ошибку зависимости во время компиляции. Я думаю, что это проблема с GRPC. Я имел в виду последнюю версию Apache Beam, но я получаю ту же ошибку.

П 

    <properties>
        <beam.version>2.8.0</beam.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-sdks-java-core</artifactId>
            <version>${beam.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
            <version>${beam.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
            <version>${beam.version}</version>
            <!--scope>runtime</scope-->
        </dependency>
        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-runners-direct-java</artifactId>
            <version>${beam.version}</version>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-firestore</artifactId>
            <version>1.0.0</version>
        </dependency>

Поток данных (Java 8) FirestoreFn: функция для вставки в пожарный магазин 

PCollection<String> pcollStorage =
                pColl
                    .apply("Read from GCS", TextIO.read().from(options.getInputFile()));
            PCollection<EventAvailabilityAlert> pcollEvent = pcollStorage
                    .apply("CsvLineToJson", ParDo.of(new CsvLineToJsonMsgFn()))
                    .apply("Firestore Insert", ParDo.of(new FirestoreFn()));
            PCollection<PubsubMessage> pcollPubsubMsg = pcollEvent
                    .apply("JsonMsgToPubsubMsg", ParDo.of(new JsonMsgToPubsubMsgFn()));
            pcollPubsubMsg
                    .apply("Sending To Pub/Sub",PubsubIO.writeMessages().to(pubSubProjectsFolder + projectId + pubSubTopicFolder + pubSubTopicName));
            pColl.run();

Ошибка Maven 

Could not resolve version conflict among [
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 -> io.grpc:grpc-core:jar:1.13.1, org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 
-> com.google.api:gax-grpc:jar:1.29.0 
-> io.grpc:grpc-protobuf:jar:1.10.1 
-> io.grpc:grpc-core:jar:1.10.1, org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 
-> com.google.api:gax-grpc:jar:1.29.0 
-> io.grpc:grpc-protobuf:jar:1.10.1 
-> io.grpc:grpc-protobuf-lite:jar:1.10.1 
-> io.grpc:grpc-core:jar:1.10.1, org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 
-> io.grpc:grpc-auth:jar:1.13.1 
-> io.grpc:grpc-core:jar:[1.13.1,1.13.1], org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 
-> io.grpc:grpc-netty:jar:1.13.1 
-> io.grpc:grpc-core:jar:[1.13.1,1.13.1], org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 
-> io.grpc:grpc-stub:jar:1.13.1 
-> io.grpc:grpc-core:jar:1.13.1, org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 
-> com.google.cloud.bigtable:bigtable-client-core:jar:1.4.0 
-> io.grpc:grpc-core:jar:1.10.1, org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 
-> io.grpc:grpc-all:jar:1.13.1 
-> io.grpc:grpc-core:jar:[1.13.1,1.13.1], org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 
-> io.grpc:grpc-all:jar:1.13.1 
-> io.grpc:grpc-okhttp:jar:1.13.1 
-> io.grpc:grpc-core:jar:[1.13.1,1.13.1], org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 
-> io.grpc:grpc-all:jar:1.13.1 
-> io.grpc:grpc-protobuf-nano:jar:1.13.1 
-> io.grpc:grpc-core:jar:1.13.1, org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 
-> io.grpc:grpc-all:jar:1.13.1 
-> io.grpc:grpc-testing:jar:1.13.1 
-> io.grpc:grpc-core:jar:[1.13.1,1.13.1], com.google.cloud:google-cloud-firestore:jar:1.0.0 
-> io.grpc:grpc-netty-shaded:jar:1.19.0 
-> io.grpc:grpc-core:jar:[1.19.0,1.19.0], com.google.cloud:google-cloud-firestore:jar:1.0.0 
-> io.opencensus:opencensus-contrib-grpc-util:jar:0.19.2 
-> io.grpc:grpc-core:jar:1.18.0]

Полная ошибка компиляции maven 

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project yyy: Could not resolve dependencies for project xx.xxx:yyy:jar:0.1: Failed to collect dependencies for xx.xxx:yyy:jar:0.1
    at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:269)
    at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
    at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)


I expect to insert the messages in real time to firestore and i think that integrating the functionality in dataflow (like a "apply" operation) is the best approach.
Does exist a **workaround** to resolve the issue or definitively is a bug of compatibility.
Could you explain if exist another eficiente design to achieve the goal (for example, calling from dataflow to firestore rest/grpc api, the last one implemented in another project).
...