Потребитель читает только от конкретного посредника в конфигурации с 3 посредниками с коэффициентом репликации 3 и разделом 1

Question

Сценарий

Я запустил все компоненты для конфигурации с несколькими брокерами на одном компьютере, используя сценарии оболочки, найденные в https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz

1. Запущен zookeeper со свойствами zookeeper
2. Запущено 3 брокера с 3 различными свойствами сервера. Они различаются только по этим значениям конфигурации

broker.id
log.dirs
port

Я также пытался изменить offsets.topic.replication.factor и transaction.state.log.replication.factor, но я не верю, что они актуальны.

Примечание: порядок, в котором я начинал брокеров, составляет 0, 1, 2

3. Создать тему с фактором репликации 3 и одним разделом

bin/kafka-topics.sh --create --topic repl_topic --zookeeper localhost:2181 --replication-factor 3 --partitions 1

4. Начинается консоль производителя и потребителя

bin/kafka-console-producer.sh --topic repl_topic --broker-list localhost:9092,localhost:9093,localhost:9094
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092,localhost:9093,localhost:9094 -topic repl_topic --from-beginning

Производитель и потребитель работают правильно. Однако, если я выключаю с помощью Ctrl-C брокера 0 (даже если он не является лидером), потребитель получает предупреждение, но больше не получает сообщения от производителя. Только когда брокер 0 снова будет в рабочем состоянии, потребитель получит все сообщения.

Мой вывод

Потребитель зависит только от брокера 0. Он не взаимодействует с другими.

Мой вопрос

Почему?

Answer 1

Я наконец-то выяснил проблему и исправил ее. Проверяя __consumer_offsets, я заметил, что он не реплицируется.

 bin/kafka-topics.sh --topic __consumer_offsets --zookeeper localhost:2181 --describe
Topic:__consumer_offsets    PartitionCount:50   ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
    Topic: __consumer_offsets   Partition: 0    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 1    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 2    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 3    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 4    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 5    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 6    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 7    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 8    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 9    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 10   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 11   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 12   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 13   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 14   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 15   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 16   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 17   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 18   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 19   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 20   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 21   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 22   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 23   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 24   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 25   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 26   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 27   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 28   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 29   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 30   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 31   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 32   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 33   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 34   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 35   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 36   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 37   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 38   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 39   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 40   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 41   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 42   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 43   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 44   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 45   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 46   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 47   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 48   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 49   Leader: 0   Replicas: 0 Isr: 0

Действительно, в первый раз, когда я начинал потребителя, это было для темы с replication_factor, равным 1. За это время потребитель создал только одну реплику. Эта тема больше не обновлялась, и тогда другие брокеры не могли ее увидеть, если брокер 0 не работал.

Решение

С помощью команды можно установить новый уровень репликации для каждого раздела. bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file new_reassignment.json --execute

Используемый там файл new_reassignment.json json имеет следующее содержимое

{"version":1,"partitions":[
{"topic":"__consumer_offsets","partition":0,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":1,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":2,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":3,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":4,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":5,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":6,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":7,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":8,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":9,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":10,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":11,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":12,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":13,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":14,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":15,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":16,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":17,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":18,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":19,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":20,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":21,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":22,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":23,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":24,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":25,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":26,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":27,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":28,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":29,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":30,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":31,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":32,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":33,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":34,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":35,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":36,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":37,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":38,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":39,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":40,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":41,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":42,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":43,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":44,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":45,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":46,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":47,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":48,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":49,"replicas":[0,1,2],"log_dirs":["any","any","any"]}]}

На этом этапе можно управлять неудачей любого брокера с успехом.

__ consumer_offsets - дополнительная информация

Количество разделов в этой конкретной теме по умолчанию установлено равным 50. Потребитель запускает его создание. Тема в основном используется потребителем для фиксации полученного сообщения для каждой темы: раздел. Если этот раздел __consumer_offsets невидим для других посредников, посредники не смогут сообщить потребителю новые сообщения.