Проблема с подключением кафки - PullRequest
Новая
       22

Проблема с подключением кафки

0 голосов
/ 21 октября 2018

Я установил Kafka на Docker Swarm, но я постоянно сталкиваюсь с проблемой ниже.

Может кто-нибудь пролить свет на проблему?а чего мне здесь не хватает? 

fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    | bundler: failed to load command: bin/subscriber_kafka (bin/subscriber_kafka)
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | I, [2018-10-21T07:37:41.204851 #1]  INFO -- : Leaving group `accounts_service.account`
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.204879 #1] DEBUG -- : Getting group coordinator for `accounts_service.account`
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/cluster.rb:121:in `get_group_coordinator'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.204927 #1] DEBUG -- : Opening connection to ip-10-0-1-109.ap-southeast-1.compute.internal:9094 with client id ruby-kafka...
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:41.204555 #1] ERROR -- : Failed to connect to ip-10-0-1-109.ap-southeast-1.compute.internal:9094: getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer_group.rb:190:in `coordinator'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.206519 #1] DEBUG -- : Closing socket to ip-10-0-1-109.ap-southeast-1.compute.internal:9094
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:41.206599 #1] ERROR -- : Failed to get coordinator info from ip-10-0-1-109.ap-southeast-1.compute.internal:9094 (node_id=109): getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer_group.rb:131:in `block in join_group'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | I, [2018-10-21T07:37:41.206633 #1]  INFO -- : Leaving group `accounts_service.customer`
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.206660 #1] DEBUG -- : Getting group coordinator for `accounts_service.customer`
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.206717 #1] DEBUG -- : Opening connection to ip-10-0-1-109.ap-southeast-1.compute.internal:9094 with client id ruby-kafka...
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.222250 #1] DEBUG -- : Handling fetcher command: stop
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/instrumenter.rb:21:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.222428 #1] DEBUG -- : Handling fetcher command: stop
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/instrumenter.rb:35:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer_group.rb:130:in `join_group'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.222955 #1] DEBUG -- : Handling fetcher command: stop
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.208336 #1] ERROR -- : Failed to connect to ip-10-0-1-109.ap-southeast-1.compute.internal:9094: getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer_group.rb:47:in `join'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:46.208412 #1] DEBUG -- : Closing socket to ip-10-0-1-109.ap-southeast-1.compute.internal:9094
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.208545 #1] ERROR -- : Failed to get coordinator info from ip-10-0-1-109.ap-southeast-1.compute.internal:9094 (node_id=109): getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:443:in `join_group'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.208265 #1] ERROR -- : Failed to connect to ip-10-0-1-109.ap-southeast-1.compute.internal:9094: getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:506:in `fetch_batches'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:46.208815 #1] DEBUG -- : Closing socket to ip-10-0-1-109.ap-southeast-1.compute.internal:9094
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.208867 #1] ERROR -- : Failed to get coordinator info from ip-10-0-1-109.ap-southeast-1.compute.internal:9094 (node_id=109): getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:206:in `block in each_message'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:394:in `block in consumer_loop'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.209073 #1] ERROR -- : Failed to connect to ip-10-0-1-109.ap-southeast-1.compute.internal:9094: getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:46.209110 #1] DEBUG -- : Closing socket to ip-10-0-1-109.ap-southeast-1.compute.internal:9094
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/instrumenter.rb:21:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.209170 #1] ERROR -- : Failed to get coordinator info from ip-10-0-1-109.ap-southeast-1.compute.internal:9094 (node_id=109): getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/instrumenter.rb:35:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | bundler: failed to load command: bin/subscriber_kafka (bin/subscriber_kafka)
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:393:in `consumer_loop'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:205:in `each_message'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/kafka_tools-0.2.0/lib/kafka_tools/consumer.rb:65:in `listen_messages'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/kafka_tools-0.2.0/lib/kafka_tools/consumer.rb:20:in `run'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/kafka_tools-0.2.0/lib/kafka_tools.rb:26:in `block (2 levels) in consume'

My Docker Swarm config 

services:
  zookeeper:
    image: wurstmeister/zookeeper:latest
    networks:
      - fastlane
    deploy:
      mode: global

  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - target: 9094
        published: 9094
        protocol: tcp
        mode: host
    environment:
      HOSTNAME_COMMAND: ${KAFKA_HOSTNAME_COMMAND}
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      BROKER_ID_COMMAND: ${BROKER_ID_COMMAND}
    networks:
      - fastlane
    depends_on:
      - zookeeper

  kafka_manager:
    image: deric/kafka-manager:1.3.3.18
    networks:
      - fastlane
    environment:
      ZK_HOSTS: zookeeper:2181
      APPLICATION_SECRET: letmein
    deploy:
      labels:
        - traefik.backend.loadbalancer.swarm=true
        - traefik.backend=kafka-manager
        - traefik.frontend.rule=Host:kafka-manager.${DOMAIN}
        - traefik.port=9000
        - traefik.docker.network=fastlane
    depends_on:
      - zookeeper
KAFKA_HOSTNAME_COMMAND = curl http://169.254.169.254/latest/meta-data/local-hostname
BROKER_ID_COMMAND="curl http://169.254.169.254/latest/meta-data/local-ipv4 | awk -F '.' '{print \$4}'"

1 Ответ

0 голосов
/ 21 октября 2018

Вы смотрели на README?В нем конкретно упоминается, что необходимо сделать для запуска в AWS

Если указано KAFKA_ADVERTISED_HOST_NAME, оно имеет приоритет над HOSTNAME_COMMAND

Для развертывания AWS вы можете использовать метаданныеСлужба для получения IP-адреса хоста контейнера:

HOSTNAME_COMMAND=wget -t3 -T2 -qO- http://169.254.169.254/latest/meta-data/local-ipv4

Ссылка: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html

Итак, сначала попробуйте то, что говорят документы, прежде чем что-то ещеи удалите пробелы вокруг знака равенства для KAFKA_HOSTNAME_COMMAND =

Если вы запускаете код Ruby локально, а не в сети AWS, вам нужно получить приведенные выше команды для возврата внешних адресов после открытия портов Kafkaиз настроек VPC в консоли AWS.

Я хотел бы отметить, что было бы проще подключиться к экземплярам Kafka, если бы ваше приложение Ruby (a) также являлось частью файла Compose.Например, используя оверлейную сеть или Træfik, вы можете получить прямой доступ к услуге kafka:9094.

...