Увеличение коэффициента репликации для темы, но все еще в процессе - PullRequest
Новая
       62

Увеличение коэффициента репликации для темы, но все еще в процессе

0 голосов
/ 26 июня 2019

мы увидели, что ReplicationFactor: 1 вместо ReplicationFactor: 3 

/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper 172.23.12.33:2181 --describe |  grep  __consumer_offsets Topic:__consumer_offsets        PartitionCount:50       ReplicationFactor:1     Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer

так, потому что мы запускаем следующую процедуру, чтобы получить ReplicationFactor: 1 вместо ReplicationFactor: 3

но когда мы бежим 

kafka-reassign-partitions --zookeeper $ZOOKEEPER_CONNECT \    --reassignment-json-file /home/liquidnt/inc-replication-factor.json  \    --verify

мы все еще в процессе !!!

это нормальное поведение? нам нужно подождать? до сих пор мы 2 часа после процедуры, но все еще видим - в процессе

1. процедура (идентификаторы брокеров - 0,1,2,3,4)

ссылка- https://gist.github.com/uarun/da30d8ef52b5d57b145cd13694c8acdc

Создание пользовательского плана переназначения (см. Вложенный файл inc-replication-factor.json). В этом случае мы идем от коэффициента репликации от 1 до 3. 

{"version":1,
 "partitions":[
   {"topic":"__consumer_offsets", "partition":0,  "replicas":[0, 1, 2]},
   {"topic":"__consumer_offsets", "partition":1,  "replicas":[0, 1, 2]},
   {"topic":"__consumer_offsets", "partition":2,  "replicas":[0, 1, 2]},
   {"topic":"__consumer_offsets", "partition":3,  "replicas":[0, 1, 2]},
   {"topic":"__consumer_offsets", "partition":4,  "replicas":[0, 1, 2]},
   {"topic":"__consumer_offsets", "partition":5,  "replicas":[0, 1, 2]},
   {"topic":"__consumer_offsets", "partition":6,  "replicas":[0, 1, 2]},
   {"topic":"__consumer_offsets", "partition":7,  "replicas":[0, 1, 2]},
   {"topic":"__consumer_offsets", "partition":8,  "replicas":[0, 1, 2]},
   {"topic":"__consumer_offsets", "partition":9,  "replicas":[0, 1, 2]},
   {"topic":"__consumer_offsets", "partition":10, "replicas":[1, 2, 3]},
   {"topic":"__consumer_offsets", "partition":11, "replicas":[1, 2, 3]},
   {"topic":"__consumer_offsets", "partition":12, "replicas":[1, 2, 3]},
   {"topic":"__consumer_offsets", "partition":13, "replicas":[1, 2, 3]},
   {"topic":"__consumer_offsets", "partition":14, "replicas":[1, 2, 3]},
   {"topic":"__consumer_offsets", "partition":15, "replicas":[1, 2, 3]},
   {"topic":"__consumer_offsets", "partition":16, "replicas":[1, 2, 3]},
   {"topic":"__consumer_offsets", "partition":17, "replicas":[1, 2, 3]},
   {"topic":"__consumer_offsets", "partition":18, "replicas":[1, 2, 3]},
   {"topic":"__consumer_offsets", "partition":19, "replicas":[1, 2, 3]},
   {"topic":"__consumer_offsets", "partition":20, "replicas":[2, 3, 4]},
   {"topic":"__consumer_offsets", "partition":21, "replicas":[2, 3, 4]},
   {"topic":"__consumer_offsets", "partition":22, "replicas":[2, 3, 4]},
   {"topic":"__consumer_offsets", "partition":23, "replicas":[2, 3, 4]},
   {"topic":"__consumer_offsets", "partition":24, "replicas":[2, 3, 4]},
   {"topic":"__consumer_offsets", "partition":25, "replicas":[2, 3, 4]},
   {"topic":"__consumer_offsets", "partition":26, "replicas":[2, 3, 4]},
   {"topic":"__consumer_offsets", "partition":27, "replicas":[2, 3, 4]},
   {"topic":"__consumer_offsets", "partition":28, "replicas":[2, 3, 4]},
   {"topic":"__consumer_offsets", "partition":29, "replicas":[2, 3, 4]},
   {"topic":"__consumer_offsets", "partition":30, "replicas":[3, 4, 0]},
   {"topic":"__consumer_offsets", "partition":31, "replicas":[3, 4, 0]},
   {"topic":"__consumer_offsets", "partition":32, "replicas":[3, 4, 0]},
   {"topic":"__consumer_offsets", "partition":33, "replicas":[3, 4, 0]},
   {"topic":"__consumer_offsets", "partition":34, "replicas":[3, 4, 0]},
   {"topic":"__consumer_offsets", "partition":35, "replicas":[3, 4, 0]},
   {"topic":"__consumer_offsets", "partition":36, "replicas":[3, 4, 0]},
   {"topic":"__consumer_offsets", "partition":37, "replicas":[3, 4, 0]},
   {"topic":"__consumer_offsets", "partition":38, "replicas":[3, 4, 0]},
   {"topic":"__consumer_offsets", "partition":39, "replicas":[3, 4, 0]},
   {"topic":"__consumer_offsets", "partition":40, "replicas":[4, 0, 1]},
   {"topic":"__consumer_offsets", "partition":41, "replicas":[4, 0, 1]},
   {"topic":"__consumer_offsets", "partition":42, "replicas":[4, 0, 1]},
   {"topic":"__consumer_offsets", "partition":43, "replicas":[4, 0, 1]},
   {"topic":"__consumer_offsets", "partition":44, "replicas":[4, 0, 1]},
   {"topic":"__consumer_offsets", "partition":45, "replicas":[4, 0, 1]},
   {"topic":"__consumer_offsets", "partition":46, "replicas":[4, 0, 1]},
   {"topic":"__consumer_offsets", "partition":47, "replicas":[4, 0, 1]},
   {"topic":"__consumer_offsets", "partition":48, "replicas":[4, 0, 1]},
   {"topic":"__consumer_offsets", "partition":49, "replicas":[4, 0, 1]}
 ]
}

2. Запустите сценарий переназначения раздела Kafka:

kafka-reassign-partitions --zookeeper $ ZOOKEEPER_CONNECT \ --reassignment-json-file /home/liquidnt/inc-replication-factor.json \ --execute

3. Проверьте, было ли назначение успешным

kafka-reassign-partitions --zookeeper $ ZOOKEEPER_CONNECT \ --reassignment-json-file /home/liquidnt/inc-replication-factor.json \ --verify

4. Больше проверок

. / Kafka-themes --zookeeper $ ZOOKEEPER_CONNECT --описать --topic __consumer_offsets

...