Neo4j процедура алгоритма графа Проблема памяти - PullRequest
Новая
       30

Neo4j процедура алгоритма графа Проблема памяти

0 голосов
/ 29 августа 2018

У меня около 32M узлов и 28M отношений. Схема выглядит следующим образом. 

(User)-[:DEVICE]->(Device) 
(User)-[:EMAIL]->(Email) 
(User)-[:SHIPPED]->(Shipzip) 
(User)-[:PHONE]->(Phone)

По сути, я хочу использовать unionFind в пакете процедур алгоритма графа для разделения узлов пользователя на группы. 

CALL algo.unionFind(
'MATCH (u1: User)
RETURN id(u1) AS id',
'MATCH (u1: User)-[:DEVICE|:EMAIL|:PHONE]->(middle)<-[:DEVICE|:EMAIL|:PHONE]-(u2:User)
WHERE id(u1) < id(u2)
RETURN id(u1) AS source, id(u2) AS target',
{graph: 'cypher', write:true, partitionProperty: "group_id_by_hard_links", concurrency:16}
)

Это процедура шифрования, которую я запустил, и кажется, что она столкнулась с некой бесконечной проблемой GC, которую GC продолжает отскакивать. 

   [Parallel Time: 6.9 ms, GC Workers: 8]
      [GC Worker Start (ms): Min: 1670347.8, Avg: 1670348.5, Max: 1670353.1, Diff: 5.4]
      [Ext Root Scanning (ms): Min: 0.0, Avg: 1.5, Max: 6.1, Diff: 6.1, Sum: 11.7]
      [Update RS (ms): Min: 0.0, Avg: 0.8, Max: 1.9, Diff: 1.9, Sum: 6.5]
         [Processed Buffers: Min: 0, Avg: 2.9, Max: 7, Diff: 7, Sum: 23]
      [Scan RS (ms): Min: 0.1, Avg: 0.3, Max: 0.6, Diff: 0.5, Sum: 2.3]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
      [Object Copy (ms): Min: 0.0, Avg: 1.7, Max: 2.4, Diff: 2.4, Sum: 13.2]
      [Termination (ms): Min: 0.0, Avg: 1.5, Max: 1.9, Diff: 1.9, Sum: 11.9]
         [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 8]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
      [GC Worker Total (ms): Min: 1.0, Avg: 5.7, Max: 6.5, Diff: 5.5, Sum: 45.7]
      [GC Worker End (ms): Min: 1670354.1, Avg: 1670354.2, Max: 1670354.3, Diff: 0.2]
   [Code Root Fixup: 0.0 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 1.4 ms]
   [Other: 3.7 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 1.4 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 1.0 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.1 ms]
      [Free CSet: 0.9 ms]
   [Eden: 13.5G(13.5G)->0.0B(13.5G) Survivors: 40.0M->32.0M Heap: 24.4G(27.3G)->10.9G(27.3G)]
 [Times: user=0.07 sys=0.00, real=0.01 secs] 
2018-08-28T14:03:30.446-0700: 1670.360: Total time for which application threads were stopped: 0.0184573 seconds, Stopping threads took: 0.0000327 seconds
2018-08-28T14:03:31.278-0700: 1671.191: Total time for which application threads were stopped: 0.0061963 seconds, Stopping threads took: 0.0000654 seconds
2018-08-28T14:03:40.972-0700: 1680.886: Total time for which application threads were stopped: 0.0063730 seconds, Stopping threads took: 0.0000550 seconds
2018-08-28T14:03:42.986-0700: 1682.899: Total time for which application threads were stopped: 0.0131774 seconds, Stopping threads took: 0.0000639 seconds
2018-08-28T14:03:45.754-0700: 1685.667: Total time for which application threads were stopped: 0.0147754 seconds, Stopping threads took: 0.0000755 seconds
2018-08-28T14:03:45.768-0700: 1685.682: Total time for which application threads were stopped: 0.0143796 seconds, Stopping threads took: 0.0000522 seconds
2018-08-28T14:03:45.791-0700: 1685.704: Total time for which application threads were stopped: 0.0102180 seconds, Stopping threads took: 0.0000482 seconds
2018-08-28T14:03:45.799-0700: 1685.712: Total time for which application threads were stopped: 0.0075455 seconds, Stopping threads took: 0.0000281 seconds

Мне интересно, есть ли способ избавиться от проблемы с памятью, чтобы завершить вызов процедуры.

Открыт для любых предложений, таких как настройка шифров или настройка GC.

...