Мы используем cassandra-3.0.15. Некоторые из запросов на чтение занимают больше времени (> 500 мс). На основании трассировки запроса кажется, что узел-координатор занимает больше времени, чем ожидалось, особенно при запуске обработки запроса. Что может быть причиной этого? Как решить эту проблему?
Приложили трассировку запроса для одного из таких запросов ниже. Видно, что первое действие «Выполнение односегментного запроса для средства_конфигурации» занимает около 1,4 с. Остальные действия выполняются довольно быстро.
activity | source | source_elapsed | thread
---------------------------------------------------------------------------+----------------+----------------+-------------------------------------------
Executing single-partition query on facility_configuration | node1 | 1466289 | SharedPool-Worker-6
Acquiring sstable references | node1 | 1466298 | SharedPool-Worker-6
Key cache hit for sstable 6 | node1 | 1466314 | SharedPool-Worker-6
Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | node1 | 1466330 | SharedPool-Worker-6
Merging data from memtables and 1 sstables | node1 | 1466341 | SharedPool-Worker-6
Read 1 live and 0 tombstone cells | node1 | 1466450 | SharedPool-Worker-6
reading digest from /node2 | node1 | 1466466 | SharedPool-Worker-6
Sending READ message to /node2 | node1 | 1467388 | MessagingService-Outgoing-/node2
READ message received from /node1 | node2 | 1 | MessagingService-Incoming-/node1
Executing single-partition query on facility_configuration | node2 | 536 | SharedPool-Worker-3
Acquiring sstable references | node2 | 546 | SharedPool-Worker-3
Key cache hit for sstable 6 | node2 | 562 | SharedPool-Worker-3
Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | node2 | 575 | SharedPool-Worker-3
Merging data from memtables and 1 sstables | node2 | 586 | SharedPool-Worker-3
Read 1 live and 0 tombstone cells | node2 | 642 | SharedPool-Worker-3
Read 1 live and 0 tombstone cells | node2 | 807 | SharedPool-Worker-3
Enqueuing response to /node1 | node2 | 819 | SharedPool-Worker-3
Sending REQUEST_RESPONSE message to /node1 | node2 | 845 | MessagingService-Outgoing-/node1
REQUEST_RESPONSE message received from /node2 | node1 | 1470583 | MessagingService-Incoming-/node2
Processing response from /node2 | node1 | 1470665 | SharedPool-Worker-15
session_id | client | command | coordinator | duration | parameters | request | started_at
--------------------------------------+--------------+---------+----------------+----------+--------------------------------------------------------------------------------------------+-----------------------------+--------------------------
ddef2100-eb53-11e9-86ed-9d59234ed651 | clientNode | QUERY | node1 | 1470765 | {'consistency_level': 'QUORUM', 'page_size': '5000', 'serial_consistency_level': 'SERIAL'} | Execute CQL3 prepared query | 2019-10-10 11:48:24+0000
facility_configuration histograms
Percentile SSTables Write Latency Read Latency Partition Size Cell Count
(micros) (micros) (bytes)
50% 1.00 0.00 182.79 124 5
75% 1.00 0.00 263.21 258 7
95% 2.00 0.00 545.79 1597 14
98% 2.00 0.00 943.13 1597 14
99% 2.00 0.00 1358.10 1597 14
Min 0.00 0.00 51.01 87 5
Max 2.00 0.00 1955.67 1597 14
tablestats
Read Count: 68390050
Read Latency: 0.08643708858233032 ms.
Write Count: 11
Write Latency: 0.09781818181818182 ms.
Pending Flushes: 0
Table: facility_configuration
SSTable count: 2
Space used (live): 12241
Space used (total): 12241
Space used by snapshots (total): 0
Off heap memory used (total): 103
SSTable Compression Ratio: 0.5963017053887002
Number of partitions (estimate): 7
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 4
Local read count: 68390066
Local read latency: 0.195 ms
Local write count: 11
Local write latency: NaN ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 40
Bloom filter off heap memory used: 24
Index summary off heap memory used: 55
Compression metadata off heap memory used: 24
Compacted partition minimum bytes: 87
Compacted partition maximum bytes: 1597
Compacted partition mean bytes: 420
Average live cells per slice (last five minutes): 1.0
Maximum live cells per slice (last five minutes): 1
Average tombstones per slice (last five minutes): 1.0
Maximum tombstones per slice (last five minutes): 1