почему в кафке 2.1.1 zstd в 5 раз МЕДЛЕННО, чем gzip?

Question

Я тестирую скорость сжатия kafka по темам, но получаю некоторые странные результаты, ZSTD в 5 раз медленнее, чем GZIP, вот некоторые результаты ..

zstd

docker exec -ti compose-single_kafka_1   bash -c '/opt/kafka/bin/kafka-producer-perf-test.sh --topic test_bench --throughput 4000000 --record-size 1000 --num-records 4000000 --producer-props bootstrap.servers="127.0.0.1:9092"   compression.type=zstd batch.size=500000'
9789 records sent, 1956.2 records/sec (1.87 MB/sec), 24.9 ms avg latency, 223.0 max latency.
11061 records sent, 2211.8 records/sec (2.11 MB/sec), 24.0 ms avg latency, 91.0 max latency.
11049 records sent, 2208.9 records/sec (2.11 MB/sec), 25.0 ms avg latency, 76.0 max latency.
11183 records sent, 2232.1 records/sec (2.13 MB/sec), 23.4 ms avg latency, 89.0 max latency.
11760 records sent, 2346.4 records/sec (2.24 MB/sec), 23.2 ms avg latency, 89.0 max latency.
11218 records sent, 2239.6 records/sec (2.14 MB/sec), 23.7 ms avg latency, 93.0 max latency.
10892 records sent, 2178.0 records/sec (2.08 MB/sec), 23.8 ms avg latency, 94.0 max latency.
...

GZIP

docker exec -ti compose-single_kafka_1   bash -c '/opt/kafka/bin/kafka-producer-perf-test.sh --topic test_bench_gz --throughput 4000000 --record-size 1000 --num-records 4000000 --producer-props bootstrap.servers="127.0.0.1:9092" compression.type=gzip batch.size=500000'
50240 records sent, 10048.0 records/sec (9.58 MB/sec), 6.6 ms avg latency, 208.0 max latency.
60849 records sent, 12109.3 records/sec (11.55 MB/sec), 5.4 ms avg latency, 55.0 max latency.
59617 records sent, 11923.4 records/sec (11.37 MB/sec), 5.6 ms avg latency, 47.0 max latency.
61800 records sent, 12360.0 records/sec (11.79 MB/sec), 5.4 ms avg latency, 50.0 max latency.
64763 records sent, 12952.6 records/sec (12.35 MB/sec), 5.1 ms avg latency, 48.0 max latency.
58560 records sent, 11712.0 records/sec (11.17 MB/sec), 5.8 ms avg latency, 48.0 max latency.
57930 records sent, 11586.0 records/sec (11.05 MB/sec), 5.8 ms avg latency, 47.0 max latency.
57360 records sent, 11472.0 records/sec (10.94 MB/sec), 5.8 ms avg latency, 48.0 max latency.
64080 records sent, 12810.9 records/sec (12.22 MB/sec), 5.2 ms avg latency, 39.0 max latency.
57120 records sent, 11401.2 records/sec (10.87 MB/sec), 5.8 ms avg latency, 83.0 max latency.

Answer 1

дальнейшее тестирование показало, что zstd, кажется, имеет более высокие фиксированные затраты, что делает его медленнее, чем gzip для небольших партий, в вышеупомянутом тесте партия была высокой (может быть, слишком высокой? Потому что что-то еще может быть введено в уравнение) вот некоторыебольше результатов ... zstd быстрее в некоторых случаях, но в других медленнее (МНОГО МЕНЬШЕ! когда включено сжатие на продюсере + повторное сжатие на брокере (вместо только для производителя я видел, что в некоторых тестах zstd против gzip в 50 раз медленнее)..)

эта вещь не задокументирована, и даже в Google никогда не упоминается ... но это ВАЖНО. Тщательно проверяйте свой вариант использования (с вашими ~ производственными данными и ~ размером партии)

 Fri Mar 08 09:51:28   @@@@on  wk1                                                                                                                                   
 ~  hist:2434  jobs:0 $docker exec -ti compose-single_kafka_client_1   bash -c '/opt/kafka/bin/kafka-producer-perf-test.sh --topic test_bench_rnd_gz --throughput 4000000 --record-size 1000 --num-records 2000000 --producer-props bootstrap.servers="kafka:9092"   compression.type=gzip batch.size=10000'
[2019-03-08 09:51:53,704] WARN [Producer clientId=producer-1] Error while fetching metadata with correlation id 1 : {test_bench_rnd_gz=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
408109 records sent, 81621.8 records/sec (77.84 MB/sec), 27.4 ms avg latency, 329.0 max latency.
491153 records sent, 98230.6 records/sec (93.68 MB/sec), 2.5 ms avg latency, 62.0 max latency.
513725 records sent, 102745.0 records/sec (97.99 MB/sec), 3.0 ms avg latency, 44.0 max latency.
514983 records sent, 102996.6 records/sec (98.23 MB/sec), 2.0 ms avg latency, 26.0 max latency.
2000000 records sent, 96679.073814 records/sec (92.20 MB/sec), 7.53 ms avg latency, 329.00 ms max latency, 2 ms 50th, 26 ms 95th, 160 ms 99th, 283 ms 99.9th.

 Fri Mar 08 09:52:14   @@@@on  wk1 
 ~  hist:2435  jobs:0 $docker exec -ti compose-single_kafka_client_1   bash -c '/opt/kafka/bin/kafka-producer-perf-test.sh --topic test_bench_rnd_zstd --throughput 4000000 --record-size 1000 --num-records 2000000 --producer-props bootstrap.servers="kafka:9092"   compression.type=zstd batch.size=10000'
569352 records sent, 113870.4 records/sec (108.60 MB/sec), 336.4 ms avg latency, 821.0 max latency.
776529 records sent, 155305.8 records/sec (148.11 MB/sec), 481.2 ms avg latency, 1120.0 max latency.
2000000 records sent, 138504.155125 records/sec (132.09 MB/sec), 410.25 ms avg latency, 1120.00 ms max latency, 406 ms 50th, 733 ms 95th, 999 ms 99th, 1102 ms 99.9th.

 Fri Mar 08 09:52:47   @@@@on  wk1 
 ~  hist:2436  jobs:0 $docker exec -ti compose-single_kafka_client_1   bash -c '/opt/kafka/bin/kafka-producer-perf-test.sh --topic test_bench_rnd_zstd --throughput 4000000 --record-size 1000 --num-records 2000000 --producer-props bootstrap.servers="kafka:9092"   compression.type=zstd batch.size=1000000'
1029044 records sent, 205808.8 records/sec (196.27 MB/sec), 166.6 ms avg latency, 354.0 max latency.
2000000 records sent, 218316.777644 records/sec (208.20 MB/sec), 148.01 ms avg latency, 354.00 ms max latency, 155 ms 50th, 264 ms 95th, 315 ms 99th, 348 ms 99.9th.

 Fri Mar 08 09:53:21   @@@@on  wk1 
 ~  hist:2437  jobs:0 $docker exec -ti compose-single_kafka_client_1   bash -c '/opt/kafka/bin/kafka-producer-perf-test.sh --topic test_bench_rnd_gz --throughput 4000000 --record-size 1000 --num-records 2000000 --producer-props bootstrap.servers="kafka:9092"   compression.type=gzip batch.size=1000000'
422704 records sent, 84540.8 records/sec (80.62 MB/sec), 1.9 ms avg latency, 195.0 max latency.
472069 records sent, 94413.8 records/sec (90.04 MB/sec), 2.7 ms avg latency, 71.0 max latency.
503248 records sent, 100649.6 records/sec (95.99 MB/sec), 2.0 ms avg latency, 41.0 max latency.
502847 records sent, 100569.4 records/sec (95.91 MB/sec), 1.6 ms avg latency, 27.0 max latency.
2000000 records sent, 94944.220271 records/sec (90.55 MB/sec), 1.99 ms avg latency, 195.00 ms max latency, 1 ms 50th, 3 ms 95th, 20 ms 99th, 58 ms 99.9th.

 Fri Mar 08 09:53:45   @@@@on  wk1 
 ~  hist:2438  jobs:0 $docker exec -ti compose-single_kafka_client_1   bash -c '/opt/kafka/bin/kafka-producer-perf-test.sh --topic test_bench_rnd_zstd --throughput 4000000 --record-size 1000 --num-records 2000000 --producer-props bootstrap.servers="kafka:9092"   compression.type=zstd batch.size=1'
3473 records sent, 686.9 records/sec (0.66 MB/sec), 1283.5 ms avg latency, 3578.0 max latency.
2662 records sent, 530.2 records/sec (0.51 MB/sec), 5465.4 ms avg latency, 7542.0 max latency.
4760 records sent, 944.3 records/sec (0.90 MB/sec), 10111.7 ms avg latency, 10959.0 max latency.
9362 records sent, 1872.4 records/sec (1.79 MB/sec), 11933.1 ms avg latency, 13012.0 max latency.
9560 records sent, 1912.0 records/sec (1.82 MB/sec), 13789.4 ms avg latency, 14587.0 max latency.
10085 records sent, 2017.0 records/sec (1.92 MB/sec), 15471.3 ms avg latency, 16141.0 max latency.
10007 records sent, 2001.4 records/sec (1.91 MB/sec), 15883.0 ms avg latency, 16088.0 max latency.
9467 records sent, 1893.4 records/sec (1.81 MB/sec), 15662.6 ms avg latency, 15754.0 max latency.
^C
 Fri Mar 08 09:54:56   @@@@on  wk1 
 ~  hist:2439  jobs:0 $docker exec -ti compose-single_kafka_client_1   bash -c '/opt/kafka/bin/kafka-producer-perf-test.sh --topic test_bench_rnd_gz --throughput 4000000 --record-size 1000 --num-records 2000000 --producer-props bootstrap.servers="kafka:9092"   compression.type=gzip batch.size=1'
17263 records sent, 3452.6 records/sec (3.29 MB/sec), 2364.6 ms avg latency, 3925.0 max latency.
25109 records sent, 5021.8 records/sec (4.79 MB/sec), 5645.0 ms avg latency, 6540.0 max latency.
26191 records sent, 5238.2 records/sec (5.00 MB/sec), 6012.7 ms avg latency, 6207.0 max latency.
25398 records sent, 5079.6 records/sec (4.84 MB/sec), 6012.9 ms avg latency, 6107.0 max latency.

хорошего дня, Франческо