У меня есть кластер из 150 узлов, Cassandra 2.2.x, 8 процессоров и 64 ГБ оперативной памяти.
Время от времени пара моих узлов имеет 100% процессора, но остальные узлы выглядят просто отлично.
Когда у меня 100% процессор:
My tpstats
Заполнение не ожидается.
INFO [Service Thread] 2018-05-12 18:03:51,636 GCInspector.java:284 - G1 Young Generation GC in 262ms. G1 Eden Space: 1291845632 -> 0; G1 Old Gen: 10237756144 -> 7451103472; G1 Survivor Space: 419430400 -> 134217728;
INFO [Service Thread] 2018-05-12 18:03:54,130 GCInspector.java:284 - G1 Young Generation GC in 286ms. G1 Eden Space: 1577058304 -> 0; G1 Old Gen: 7451103472 -> 4596956920; G1 Survivor Space: 134217728 -> 150994944;
INFO [Service Thread] 2018-05-12 18:03:57,652 GCInspector.java:284 - G1 Young Generation GC in 213ms. G1 Eden Space: 1560281088 -> 0; G1 Old Gen: 4596956920 -> 4135583472; G1 Survivor Space: 150994944 -> 134217728;
INFO [CompactionExecutor:172351] 2018-05-12 18:21:29,098 AutoSavingCache.java:399 - Saved CounterCache (258640 items) in 298 ms
INFO [CompactionExecutor:172352] 2018-05-12 18:21:29,979 AutoSavingCache.java:399 - Saved KeyCache (788252 items) in 1282 ms
В другие дни это не тот же самый узел, у меня есть другие узлы, у которых 100% ЦП, есть идеи?
Размер кучи Изображение
Table: abc
SSTable count: 14
Space used (live): 3 960 499 258
Space used (total): 3960499258
Space used by snapshots (total): 0
Off heap memory used (total): 8427312
SSTable Compression Ratio: 0.25806487155056185
Number of keys (estimate): 379760
Memtable cell count: 172086
Memtable data size: 1253025
Memtable off heap memory used: 4654441
Memtable switch count: 192
Local read count: 33923975
Local read latency: 15.081 ms
Local write count: 30595833
Local write latency: 0.144 ms
Pending flushes: 0
Bloom filter false positives: 18478
Bloom filter false ratio: 0.00000
Bloom filter space used: 1523216
Bloom filter off heap memory used: 1523104
Index summary off heap memory used: 459327
Compression metadata off heap memory used: 1790440
Compacted partition minimum bytes: 30
Compacted partition maximum bytes: 4055269
Compacted partition mean bytes: 11408
Average live cells per slice (last five minutes): 1.0
Maximum live cells per slice (last five minutes): 1
Average tombstones per slice (last five minutes): 178.40981886703793
Maximum tombstones per slice (last five minutes): 258
Table: xyz
SSTable count: 8
Space used (live): 2034316302
Space used (total): 2034316302
Space used by snapshots (total): 0
Off heap memory used (total): 4289964
SSTable Compression Ratio: 0.24435625500089553
Number of keys (estimate): 260117
Memtable cell count: 133822
Memtable data size: 832669
Memtable off heap memory used: 2215708
Memtable switch count: 175
Local read count: 10034316
Local read latency: 11.653 ms
Local write count: 27984185
Local write latency: 0.075 ms
Pending flushes: 0
Bloom filter false positives: 261009
Bloom filter false ratio: 0.04762
Bloom filter space used: 1007024
Bloom filter off heap memory used: 1006960
Index summary off heap memory used: 200632
Compression metadata off heap memory used: 866664
Compacted partition minimum bytes: 61
Compacted partition maximum bytes: 5839588
Compacted partition mean bytes: 8883
Average live cells per slice (last five minutes): 1.0
Maximum live cells per slice (last five minutes): 1
Average tombstones per slice (last five minutes): 28.292941014323677
Maximum tombstones per slice (last five minutes): 60
Table: zzz
SSTable count: 8
Space used (live): 1542111866
Space used (total): 1542111866
Space used by snapshots (total): 0
Off heap memory used (total): 14022908
SSTable Compression Ratio: 0.5677972573793397
Number of keys (estimate): 8670916
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 10623352
Bloom filter off heap memory used: 10623288
Index summary off heap memory used: 3157452
Compression metadata off heap memory used: 242168
Compacted partition minimum bytes: 36
Compacted partition maximum bytes: 268650950
Compacted partition mean bytes: 242
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Теперь нагрузка упала. Просто для сравнения после загрузки abc таблицы stats:
Table: abc
SSTable count: 14
Space used (live): 3960499258
Space used (total): 3960499258
Space used by snapshots (total): 0
Off heap memory used (total): 24792685
SSTable Compression Ratio: 0.25806487155056185
Number of keys (estimate): 380267
Memtable cell count: 780522
Memtable data size: 3746628
Memtable off heap memory used: 21019814
Memtable switch count: 192
Local read count: 33927339
Local read latency: 9.319 ms
Local write count: 30598848
Local write latency: 0.093 ms
Pending flushes: 0
Bloom filter false positives: 18478
Bloom filter false ratio: 0.00000
Bloom filter space used: 1523216
Bloom filter off heap memory used: 1523104
Index summary off heap memory used: 459327
Compression metadata off heap memory used: 1790440
Compacted partition minimum bytes: 30
Compacted partition maximum bytes: 4055269
Compacted partition mean bytes: 11408
Average live cells per slice (last five minutes): 1.0
Maximum live cells per slice (last five minutes): 1
Average tombstones per slice (last five minutes): 179.0
Maximum tombstones per slice (last five minutes): 179