Я пытался использовать пряжу для запуска OLAP и массовой загрузки. Версия janusgrapg-0.3.2, spark-2.2.1
Чтобы запустить spark-пряжу в janusgraph, я копирую банки spark в $ JANUSGRAPH_HOME / lib.
hadoop-graphson.properties:
gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
gremlin.hadoop.graphReader=org.apache.tinkerpop.gremlin.hadoop.structure.io.graphson.GraphSONInputFormat
gremlin.hadoop.graphWriter=org.apache.tinkerpop.gremlin.hadoop.structure.io.graphson.GraphSONOutputFormat
gremlin.hadoop.inputLocation=data/tinkerpop-modern.json
gremlin.hadoop.outputLocation=output
gremlin.hadoop.jarsInDistributedCache=true
gremlin.vertexProgram=org.apache.tinkerpop.gremlin.process.computer.ranking.pagerank.PageRankVertexProgram
giraph.minWorkers=2
giraph.maxWorkers=2
spark.master=yarn
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator=org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoRegistrator
команда в консоли gremlin:
graph = GraphFactory.open('/Users/lwh/dev/janusgraph-0.3.2-SNAPSHOT-hadoop2/conf/hadoop-graph/hadoop-graphson.properties')
blvp = BulkLoaderVertexProgram.build().bulkLoader(OneTimeBulkLoader).writeGraph('/Users/lwh/dev/janusgraph-0.3.2-SNAPSHOT-hadoop2/conf/janusgraph-cql-es.properties').create(graph);graph.compute(SparkGraphComputer).program(blvp).submit().get()
результат в пряже:
Application application_1547728549460_0008 failed 2 times due to AM Container for appattempt_1547728549460_0008_000002 exited with exitCode: -1000
For more detailed output, check application tracking page:http://lwhdeMacBook-Pro.local:8088/cluster/app/application_1547728549460_0008Then, click on links to logs of each attempt.
Diagnostics: Resource file:/Users/lwh/.sparkStaging/application_1547728549460_0008/__spark_libs__7746382379788050915.zip changed on src filesystem (expected 1547732288000, was 1547732288958
java.io.IOException: Resource file:/Users/lwh/.sparkStaging/application_1547728549460_0008/__spark_libs__7746382379788050915.zip changed on src filesystem (expected 1547732288000, was 1547732288958
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:255)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Failing this attempt. Failing the application.