Ошибка запуска нескольких брокеров в Apache Kafka - - PullRequest
Новая
       3

Ошибка запуска нескольких брокеров в Apache Kafka -

0 голосов
/ 28 июня 2019

Я нуб в Apache Kafka.Я читаю книгу Краткое руководство по Apache Kafka и следую инструкциям по настройке среды.Итак, я с успехом приступил к Платформе Confluent (Kafka плюс Zookeeper).Далее я пытаюсь инициализировать несколько брокеров, но происходит ошибка при запуске broker.id = 2.Я сделал следующие шаги:

1 - я создал папку с config / server.properties с именем
mark-1.properties и mark-2.properties

2 - Далее,я написал в mark-1.properties и
mark-2.properties:

    broker.id=1
    port=9093
    log.dirs=/tmp/mark-1-logs

    broker.id=2
    port=9094
    log.dirs=/tmp/mark-2-logs

3 - С Confluent уже запущен, я набрал в командной строке: 

    ./bin/kafka-server-start ./mark/mark-1.properties &

     and in other command line window

    ./bin/kafka-server-start ./mark/mark-2.properties &

Ошибка вывода: 

    [2019-06-28 00:35:26,535] ERROR Fatal error during KafkaServer     
    startup. Prepare to shutdown (kafka.server.KafkaServer)
    kafka.common.InconsistentBrokerIdException: Configured broker.id 2     
    doesn't match stored broker.id 1 in meta.properties. If you moved 
    your data, make sure your configured broker.id matches. If you 
    intend to create a new broker, you should remove all data in your 
    data directories (log.dirs).

Я выполнил все инструкции книги и foruns, но ошибка сохраняется.Что я могу сделать?

Я ожидаю, что несколько брокеров будут работать независимо друг от друга.

Все ошибки журнала: 

    [2019-06-28 15:06:51,018] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2019-06-28 15:06:52,717] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2019-06-28 15:06:52,721] INFO starting (kafka.server.KafkaServer)
[2019-06-28 15:06:52,722] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2019-06-28 15:06:52,753] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2019-06-28 15:06:52,768] INFO Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,768] INFO Client environment:host.name=ip-172-31-72-19.ec2.internal (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,768] INFO Client environment:java.version=11.0.3 (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,768] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,768] INFO Client environment:java.home=/usr/lib/jvm/java-11-openjdk-amd64 (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,769] INFO Client environment:java.class.path=/opt/kafka/kafka_2.12-2.3.0/bin/../libs/activation-1.1.1.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/aopalliance-repackaged-2.5.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/connect-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/connect-basic-auth-extension-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/connect-file-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/connect-json-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/connect-runtime-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/connect-transforms-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/guava-20.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/hk2-api-2.5.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/hk2-locator-2.5.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/hk2-utils-2.5.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-annotations-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-core-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-databind-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-dataformat-csv-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-datatype-jdk8-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-jaxrs-base-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-jaxrs-json-provider-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-module-jaxb-annotations-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-module-paranamer-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-module-scala_2.12-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jakarta.annotation-api-1.3.4.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jakarta.inject-2.5.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jersey-client-2.28.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jersey-common-2.28.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jersey-container-servlet-2.28.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jersey-container-servlet-core-2.28.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jersey-hk2-2.28.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jersey-media-jaxb-2.28.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jersey-server-2.28.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-client-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-continuation-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-http-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-io-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-security-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-server-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-servlet-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-servlets-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-util-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jsr305-3.0.2.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka-clients-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka-log4j-appender-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka-streams-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka-streams-examples-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka-streams-scala_2.12-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka-streams-test-utils-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka-tools-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka_2.12-2.3.0-sources.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka_2.12-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/log4j-1.2.17.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/lz4-java-1.6.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/maven-artifact-3.6.1.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/paranamer-2.8.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/plexus-utils-3.2.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/reflections-0.9.11.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/rocksdbjni-5.18.3.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/scala-library-2.12.8.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/scala-logging_2.12-3.9.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/scala-reflect-2.12.8.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/slf4j-api-1.7.26.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/slf4j-log4j12-1.7.26.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/snappy-java-1.1.7.3.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/spotbugs-annotations-3.1.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/validation-api-2.0.1.Final.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/zkclient-0.11.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/zookeeper-3.4.14.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/zstd-jni-1.4.0-1.jar (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,771] INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,771] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,772] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,772] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,772] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,772] INFO Client environment:os.version=4.15.0-1041-aws (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,772] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,772] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,773] INFO Client environment:user.dir=/opt/kafka/kafka_2.12-2.3.0 (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,774] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@58e1d9d (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,819] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2019-06-28 15:06:52,862] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2019-06-28 15:06:52,876] INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2019-06-28 15:06:52,887] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100000ec2640001, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2019-06-28 15:06:52,910] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2019-06-28 15:06:53,309] INFO Cluster ID = LM8y5RcXQJCMJeIJlmi34A (kafka.server.KafkaServer)
[2019-06-28 15:06:53,329] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentBrokerIdException: Configured broker.id 2 doesn't match stored broker.id 1 in meta.properties. If you moved your data, make sure your configured broker.id matches. If you intend to create a new broker, you should remove all data in your data directories (log.dirs).
    at kafka.server.KafkaServer.getBrokerIdAndOfflineDirs(KafkaServer.scala:715)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:214)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
    at kafka.Kafka$.main(Kafka.scala:84)
    at kafka.Kafka.main(Kafka.scala)
[2019-06-28 15:06:53,333] INFO shutting down (kafka.server.KafkaServer)
[2019-06-28 15:06:53,350] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2019-06-28 15:06:53,365] INFO EventThread shut down for session: 0x100000ec2640001 (org.apache.zookeeper.ClientCnxn)
[2019-06-28 15:06:53,387] INFO Session: 0x100000ec2640001 closed (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:53,393] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2019-06-28 15:06:53,423] INFO shut down completed (kafka.server.KafkaServer)
[2019-06-28 15:06:53,427] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2019-06-28 15:06:53,479] INFO shutting down (kafka.server.KafkaServer)

Ответы [ 2 ]

0 голосов
/ 28 июня 2019

Уважаемые, ошибка была моей.Я писал дубликаты строк с командами о порте и log.dir в свойствах конфигурации новых брокеров kafka.

Спасибо за внимание!

Ниже приведен фрагмент правильного файла конфигурации: 

    # The id of the broker. This must be set to a unique integer for each broker.
broker.id=1

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from 
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9093

# Hostname and port the broker will advertise to producers and consumers. If not set, 
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092

# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3

# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600


############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/tmp/kafka-logs-1
0 голосов
/ 28 июня 2019

Вы начали с того же самого файла ./bin/Kafka-server-start?

Если это так, вы должны скопировать все файлы kafka в другой каталог.

В том же каталоге запуска невозможноуказать только разные конфигурационные файлы запуска кафки.

Добро пожаловать на сайт PullRequest, где вы можете задавать вопросы и получать ответы от других членов сообщества.

Нет похожих вопросов

...