Кафка и зоопарк не будут стабильно соединяться в AWS ECS

Question

Я пытаюсь запустить контейнеры kafka и zookeeper в AWS ECS, но изо всех сил пытаюсь привести их в стабильное состояние.

Zookeeper - я использую официальный образ zookeeper: 3.5, а сервис обнаружения сервисов ECS дает ему IP-адрес по адресу zk0.ecs.demo.com

Кафка - я использую последнюю версию образа wurstmeister / kafka, и у меня есть служба обнаружения служб ECS, которая дает ему IP-адрес по адресу kafka1.ecs.demo.com настройка env var KAFKA_ZOOKEEPER_CONNECT для просмотра zk0.ecs.demo.com, а KAFKA_ADVERTISED_HOST_NAME - kafka1.ecs.demo.com:9092

Zookeeper запускается и, кажется, работает стабильно, kafka запускается и, как только он распознает zk0.ecs.demo.com в качестве хоста, он подключается, вызывая некоторые ошибки в журналах zookeeper.

Кажется, что они соединяются друг с другом, пока примерно через минуту не умрет zookeeper, а затем kafka не сможет найти его по старому IP-адресу и умрет, и все это происходит по кругу.

Сеть широко открыта со всем разрешенным трафиком, поэтому я не уверен, что это проблема.

Я пытался запустить их на машинах t2.large с памятью 2048, поэтому я не уверен, что это проблема с ресурсами.

я пробовал - работает ZK как кластер из 3 - сопоставление портов, сопоставленных с узлами и не сопоставленных - использование IP-адреса экземпляра ECS EC2 вместо частного dns

Не могу заставить его работать.

Я не рассматривал скрипт terraform, который я использую, для одного, если кто-то хочет попытаться воссоздать его. (потребуется ключ ssh в ~ / .ssh / id_rsa.pub) https://gist.github.com/chestercodes/480dda384229adbd75fefea8fdf483d4

ЗК бревна

15:53:03 ZooKeeper JMX enabled by default
15:53:04 Using config: /conf/zoo.cfg
53:04,044 [myid:] - INFO [main:QuorumPeerConfig@117] - Reading configuration from: /conf/zoo.cfg
53:04,051 [myid:] - INFO [main:QuorumPeerConfig@317] - clientPort is not set
53:04,051 [myid:] - INFO [main:QuorumPeerConfig@331] - secureClientPort is not set
53:04,058 [myid:] - WARN [main:QuorumPeerConfig@590] - No server failure will be tolerated. You need at least 3 servers.
53:04,072 INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
53:04,072 INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
53:04,072 INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled.
53:04,073 INFO [main:ManagedUtil@46] - Log4j found with jmx enabled.
53:04,085 INFO [main:QuorumPeerMain@138] - Starting quorum peer
53:04,104 INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 4 worker threads, and 64 kB direct buffers.
53:04,112 INFO [main:NIOServerCnxnFactory@686] - binding to port /0.0.0.0:2181
53:04,149 INFO [main:Log@186] - Logging initialized @753ms
53:04,204 WARN [main:ContextHandler@1339] - o.e.j.s.ServletContextHandler@8bd1b6a{/,null,null} contextPath ends with /*
53:04,204 WARN [main:ContextHandler@1350] - Empty contextPath
53:04,225 INFO [main:QuorumPeer@1349] - Local sessions disabled
53:04,225 INFO [main:QuorumPeer@1360] - Local session upgrading disabled
53:04,226 INFO [main:QuorumPeer@1327] - tickTime set to 2000
53:04,227 INFO [main:QuorumPeer@1371] - minSessionTimeout set to 4000
53:04,227 INFO [main:QuorumPeer@1382] - maxSessionTimeout set to 40000
53:04,227 INFO [main:QuorumPeer@1397] - initLimit set to 5
53:04,243 INFO [main:FileTxnSnapLog@320] - Snapshotting: 0x0 to /data/version-2/snapshot.0
53:04,246 INFO [main:QuorumPeer@798] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
53:04,249 INFO [main:QuorumPeer@813] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
53:04,256 INFO [main:Server@327] - jetty-9.2.18.v20160721
53:04,289 INFO [main:ContextHandler@744] - Started o.e.j.s.ServletContextHandler@8bd1b6a{/,null,AVAILABLE}
53:04,311 INFO [main:AbstractConnector@266] - Started ServerConnector@14ab98c3{HTTP/1.1}{0.0.0.0:8080}
53:04,312 INFO [main:Server@379] - Started @921ms
53:04,312 INFO [main:JettyAdminServer@112] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands
53:04,319 INFO [QuorumPeerListener:QuorumCnxManager$Listener@636] - My election bind port: localhost/127.0.0.1:3888
53:04,323 INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumPeer@1055] - LOOKING
53:04,324 INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@894] - New election. My id = 1, proposed zxid=0x0
53:04,326 INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
53:04,527 INFO [...)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=LeaderElection]
53:04,528 INFO [...)(secure=disabled):QuorumPeer@1143] - LEADING
53:04,531 INFO [...)(secure=disabled):Leader@63] - TCP NoDelay set to: true
53:04,531 INFO [...)(secure=disabled):Leader@83] - zookeeper.leader.maxConcurrentSnapshots = 10
53:04,532 INFO [...)(secure=disabled):Leader@85] - zookeeper.leader.maxConcurrentSnapshotTimeout = 5
53:04,538 INFO [...)(secure=disabled):Environment@109] - Server environment:zookeeper.version=3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60, built on 04/03/2017 16:19 GMT
53:04,539 INFO [...)(secure=disabled):Environment@109] - Server environment:host.name=ip-10-0-1-117.eu-west-1.compute.internal
53:04,539 INFO [...] - Server environment:java.version=1.8.0_151
53:04,539 INFO [...] - Server environment:java.vendor=Oracle Corporation
53:04,539 INFO [...] - Server environment:java.home=/usr/lib/jvm/java-1.8-openjdk/jre
53:04,539 INFO [...] - Server environment:java.class.path=/zookeeper-3.5.3-beta/bin/../build/classes:/zookeeper-3.5.3-beta/bin/../build/lib/*.jar:/zookeeper-3.5.3-beta/bin/../lib/slf4j-log4j12-1.7.5.jar:/zookeeper-3.5.3-beta/bin/../lib/slf4j-api-1.7.5.jar:/zookeeper-3.5.3-beta/bin/../lib/netty-3.10.5.Fin
53:04,539 INFO [...] - Server environment:java.library.path=/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
53:04,539 INFO [...] - Server environment:java.io.tmpdir=/tmp
53:04,539 INFO [...] - Server environment:java.compiler=<NA>
53:04,539 INFO [...] - Server environment:os.name=Linux
53:04,540 INFO [...] - Server environment:os.arch=amd64
53:04,540 INFO [...] - Server environment:os.version=4.9.85-38.58.amzn1.x86_64
53:04,540 INFO [...] - Server environment:user.name=zookeeper
53:04,540 INFO [...] - Server environment:user.home=/home/zookeeper
53:04,540 INFO [...] - Server environment:user.dir=/zookeeper-3.5.3-beta
53:04,540 INFO [...] - Server environment:os.memory.free=53MB
53:04,540 INFO [...] - Server environment:os.memory.max=889MB
53:04,540 INFO [...] - Server environment:os.memory.total=59MB
53:04,541 INFO [...:ZooKeeperServer@907] - minSessionTimeout set to 4000
53:04,541 INFO [...:ZooKeeperServer@916] - maxSessionTimeout set to 40000
53:04,542 INFO [...:ZooKeeperServer@159] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /datalog/version-2 snapdir /data/version-2
53:04,543 INFO [...:Leader@414] - LEADING - LEADER ELECTION TOOK - 15 MS
53:04,547 INFO [...:FileTxnSnapLog@320] - Snapshotting: 0x0 to /data/version-2/snapshot.0
53:04,564 INFO [...:Leader@1258] - Have quorum of supporters, sids: [ [1],[1] ]; starting up and setting last processed zxid: 0x100000000
53:04,578 INFO [...:CommitProcessor@255] - Configuring CommitProcessor with 2 worker threads.
53:04,587 INFO [...:ContainerManager@64] - Using checkIntervalMs=60000 maxPerMinute=10000
54:21,619 INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.0.1.91:44016
54:21,627 INFO [NIOWorkerThread-1:ZooKeeperServer@1013] - Client attempting to establish new session at /10.0.1.91:44016
54:21,630 INFO [SyncThread:1:FileTxnLog@204] - Creating new log file: log.100000001
54:21,638 INFO [CommitProcWorkThread-1:ZooKeeperServer@727] - Established session 0x1000001141b0000 with negotiated timeout 6000 for client /10.0.1.91:44016
54:21,708 INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@880] - Got user-level KeeperException when processing sessionid:0x1000001141b0000 type:create cxid:0x2 zxid:0x100000003 txntype:-1 reqpath:n/a Error Path:/brokers Error:KeeperErrorCode = NoNode for /brokers
54:21,725 INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@880] - Got user-level KeeperException when processing sessionid:0x1000001141b0000 type:create cxid:0x6 zxid:0x100000007 txntype:-1 reqpath:n/a Error Path:/config Error:KeeperErrorCode = NoNode for /config
54:21,733 INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@880] - Got user-level KeeperException when processing sessionid:0x1000001141b0000 type:create cxid:0x9 zxid:0x10000000a txntype:-1 reqpath:n/a Error Path:/admin Error:KeeperErrorCode = NoNode for /admin
54:22,012 INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@880] - Got user-level KeeperException when processing sessionid:0x1000001141b0000 type:create cxid:0x15 zxid:0x100000015 txntype:-1 reqpath:n/a Error Path:/cluster Error:KeeperErrorCode = NoNode for /cluster
54:22,818 INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@880] - Got user-level KeeperException when processing sessionid:0x1000001141b0000 type:setData cxid:0x22 zxid:0x10000001b txntype:-1 reqpath:n/a Error Path:/controller_epoch Error:KeeperErrorCode = NoNode for /controller_epoch
54:23,031 INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@880] - Got user-level KeeperException when processing sessionid:0x1000001141b0000 type:delete cxid:0x32 zxid:0x10000001e txntype:-1 reqpath:n/a Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election

Кафка бревна

15:54:22 log.roll.hours = 168
15:54:22 log.roll.jitter.hours = 0
15:54:22 log.roll.jitter.ms = null
15:54:22 log.roll.ms = null
15:54:22 log.segment.bytes = 1073741824
15:54:22 log.segment.delete.delay.ms = 60000
15:54:22 max.connections.per.ip = 2147483647
15:54:22 max.connections.per.ip.overrides =
15:54:22 max.incremental.fetch.session.cache.slots = 1000
15:54:22 message.max.bytes = 1000012
15:54:22 metric.reporters = []
15:54:22 metrics.num.samples = 2
15:54:22 metrics.recording.level = INFO
15:54:22 metrics.sample.window.ms = 30000
15:54:22 min.insync.replicas = 1
15:54:22 num.io.threads = 8
15:54:22 num.network.threads = 3
15:54:22 num.partitions = 1
15:54:22 num.recovery.threads.per.data.dir = 1
15:54:22 num.replica.alter.log.dirs.threads = null
15:54:22 num.replica.fetchers = 1
15:54:22 offset.metadata.max.bytes = 4096
15:54:22 offsets.commit.required.acks = -1
15:54:22 offsets.commit.timeout.ms = 5000
15:54:22 offsets.load.buffer.size = 5242880
15:54:22 offsets.retention.check.interval.ms = 600000
15:54:22 offsets.retention.minutes = 1440
15:54:22 offsets.topic.compression.codec = 0
15:54:22 offsets.topic.num.partitions = 50
15:54:22 offsets.topic.replication.factor = 1
15:54:22 offsets.topic.segment.bytes = 104857600
15:54:22 password.encoder.cipher.algorithm = AES/CBC/PKCS5Padding
15:54:22 password.encoder.iterations = 4096
15:54:22 password.encoder.key.length = 128
15:54:22 password.encoder.keyfactory.algorithm = null
15:54:22 password.encoder.old.secret = null
15:54:22 password.encoder.secret = null
15:54:22 port = 9092
15:54:22 principal.builder.class = null
15:54:22 producer.purgatory.purge.interval.requests = 1000
15:54:22 queued.max.request.bytes = -1
15:54:22 queued.max.requests = 500
15:54:22 quota.consumer.default = 9223372036854775807
15:54:22 quota.producer.default = 9223372036854775807
15:54:22 quota.window.num = 11
15:54:22 quota.window.size.seconds = 1
15:54:22 replica.fetch.backoff.ms = 1000
15:54:22 replica.fetch.max.bytes = 1048576
15:54:22 replica.fetch.min.bytes = 1
15:54:22 replica.fetch.response.max.bytes = 10485760
15:54:22 replica.fetch.wait.max.ms = 500
15:54:22 replica.high.watermark.checkpoint.interval.ms = 5000
15:54:22 replica.lag.time.max.ms = 10000
15:54:22 replica.socket.receive.buffer.bytes = 65536
15:54:22 replica.socket.timeout.ms = 30000
15:54:22 replication.quota.window.num = 11
15:54:22 replication.quota.window.size.seconds = 1
15:54:22 request.timeout.ms = 30000
15:54:22 reserved.broker.max.id = 1000
15:54:22 sasl.enabled.mechanisms = [GSSAPI]
15:54:22 sasl.jaas.config = null
15:54:22 sasl.kerberos.kinit.cmd = /usr/bin/kinit
15:54:22 sasl.kerberos.min.time.before.relogin = 60000
15:54:22 sasl.kerberos.principal.to.local.rules = [DEFAULT]
15:54:22 sasl.kerberos.service.name = null
15:54:22 sasl.kerberos.ticket.renew.jitter = 0.05
15:54:22 sasl.kerberos.ticket.renew.window.factor = 0.8
15:54:22 sasl.mechanism.inter.broker.protocol = GSSAPI
15:54:22 security.inter.broker.protocol = PLAINTEXT
15:54:22 socket.receive.buffer.bytes = 102400
15:54:22 socket.request.max.bytes = 104857600
15:54:22 socket.send.buffer.bytes = 102400
15:54:22 ssl.cipher.suites = []
15:54:22 ssl.client.auth = none
15:54:22 ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
15:54:22 ssl.endpoint.identification.algorithm = null
15:54:22 ssl.key.password = null
15:54:22 ssl.keymanager.algorithm = SunX509
15:54:22 ssl.keystore.location = null
15:54:22 ssl.keystore.password = null
15:54:22 ssl.keystore.type = JKS
15:54:22 ssl.protocol = TLS
15:54:22 ssl.provider = null
15:54:22 ssl.secure.random.implementation = null
15:54:22 ssl.trustmanager.algorithm = PKIX
15:54:22 ssl.truststore.location = null
15:54:22 ssl.truststore.password = null
15:54:22 ssl.truststore.type = JKS
15:54:22 transaction.abort.timed.out.transaction.cleanup.interval.ms = 60000
15:54:22 transaction.max.timeout.ms = 900000
15:54:22 transaction.remove.expired.transaction.cleanup.interval.ms = 3600000
15:54:22 transaction.state.log.load.buffer.size = 5242880
15:54:22 transaction.state.log.min.isr = 1
15:54:22 transaction.state.log.num.partitions = 50
15:54:22 transaction.state.log.replication.factor = 1
15:54:22 transaction.state.log.segment.bytes = 104857600
15:54:22 transactional.id.expiration.ms = 604800000
15:54:22 unclean.leader.election.enable = false
15:54:22 zookeeper.connect = zk0.ecs.demo.com:2181
15:54:22 zookeeper.connection.timeout.ms = 6000
15:54:22 zookeeper.max.in.flight.requests = 10
15:54:22 zookeeper.session.timeout.ms = 6000
15:54:22 zookeeper.set.acl = false
15:54:22 zookeeper.sync.time.ms = 2000
15:54:22 (kafka.server.KafkaConfig)
15:54:22 [2018-05-04 15:54:22,162] INFO [ThrottledRequestReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
15:54:22 [2018-05-04 15:54:22,162] INFO [ThrottledRequestReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
15:54:22 [2018-05-04 15:54:22,163] INFO [ThrottledRequestReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
15:54:22 [2018-05-04 15:54:22,198] INFO Log directory '/kafka/kafka-logs-ip-10-0-1-91.eu-west-1.compute.internal' not found, creating it. (kafka.log.LogManager)
15:54:22 [2018-05-04 15:54:22,209] INFO Loading logs. (kafka.log.LogManager)
15:54:22 [2018-05-04 15:54:22,221] INFO Logs loading complete in 12 ms. (kafka.log.LogManager)
15:54:22 [2018-05-04 15:54:22,234] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
15:54:22 [2018-05-04 15:54:22,239] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
15:54:22 [2018-05-04 15:54:22,611] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
15:54:22 [2018-05-04 15:54:22,642] INFO [SocketServer brokerId=1001] Started 1 acceptor threads (kafka.network.SocketServer)
15:54:22 [2018-05-04 15:54:22,669] INFO [ExpirationReaper-1001-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
15:54:22 [2018-05-04 15:54:22,674] INFO [ExpirationReaper-1001-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
15:54:22 [2018-05-04 15:54:22,674] INFO [ExpirationReaper-1001-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
15:54:22 [2018-05-04 15:54:22,692] INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler)
15:54:22 [2018-05-04 15:54:22,713] INFO Creating /brokers/ids/1001 (is it secure? false) (kafka.zk.KafkaZkClient)
15:54:22 [2018-05-04 15:54:22,717] INFO Result of znode creation at /brokers/ids/1001 is: OK (kafka.zk.KafkaZkClient)
15:54:22 [2018-05-04 15:54:22,718] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: ArrayBuffer(EndPoint(kafka1.ecs.demo.com:9092,9092,ListenerName(PLAINTEXT),PLAINTEXT)) (kafka.zk.KafkaZkClient)
15:54:22 [2018-05-04 15:54:22,720] WARN No meta.properties file under dir /kafka/kafka-logs-ip-10-0-1-91.eu-west-1.compute.internal/meta.properties (kafka.server.BrokerMetadataCheckpoint)
15:54:22 [2018-05-04 15:54:22,789] INFO [ExpirationReaper-1001-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
15:54:22 [2018-05-04 15:54:22,789] INFO [ExpirationReaper-1001-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
15:54:22 [2018-05-04 15:54:22,801] INFO Creating /controller (is it secure? false) (kafka.zk.KafkaZkClient)
15:54:22 [2018-05-04 15:54:22,801] INFO [ExpirationReaper-1001-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
15:54:22 [2018-05-04 15:54:22,813] INFO Result of znode creation at /controller is: OK (kafka.zk.KafkaZkClient)
15:54:22 [2018-05-04 15:54:22,828] INFO [GroupCoordinator 1001]: Starting up. (kafka.coordinator.group.GroupCoordinator)
15:54:22 [2018-05-04 15:54:22,829] INFO [GroupCoordinator 1001]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
15:54:22 [2018-05-04 15:54:22,838] INFO [GroupMetadataManager brokerId=1001] Removed 0 expired offsets in 8 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
15:54:22 [2018-05-04 15:54:22,859] INFO [ProducerId Manager 1001]: Acquired new producerId block (brokerId:1001,blockStartProducerId:0,blockEndProducerId:999) by writing to Zk with path version 1 (kafka.coordinator.transaction.ProducerIdManager)
15:54:22 [2018-05-04 15:54:22,929] INFO [TransactionCoordinator id=1001] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
15:54:22 [2018-05-04 15:54:22,936] INFO [TransactionCoordinator id=1001] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
15:54:22 [2018-05-04 15:54:22,937] INFO [Transaction Marker Channel Manager 1001]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
15:54:23 [2018-05-04 15:54:23,047] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
15:54:23 [2018-05-04 15:54:23,070] INFO Kafka version : 1.1.0 (org.apache.kafka.common.utils.AppInfoParser)
15:54:23 [2018-05-04 15:54:23,070] INFO Kafka commitId : fdcf75ea326b8e07 (org.apache.kafka.common.utils.AppInfoParser)
15:54:23 [2018-05-04 15:54:23,071] INFO [KafkaServer id=1001] started (kafka.server.KafkaServer)
15:56:37 [2018-05-04 15:56:37,770] INFO Unable to read additional data from server sessionid 0x1000001141b0000, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
15:56:39 [2018-05-04 15:56:39,772] INFO Opening socket connection to server ip-10-0-1-117.eu-west-1.compute.internal/10.0.1.117:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
15:56:40 [2018-05-04 15:56:40,798] WARN Session 0x1000001141b0000 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
15:56:40 java.net.ConnectException: Connection refused
15:56:40 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
15:56:40 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
15:56:40 at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
15:56:40 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
15:56:42 [2018-05-04 15:56:42,475] INFO Opening socket connection to server ip-10-0-1-117.eu-west-1.compute.internal/10.0.1.117:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
15:56:48 [2018-05-04 15:56:48,482] WARN Client session timed out, have not heard from server in 7583ms for sessionid 0x1000001141b0000 (org.apache.zookeeper.ClientCnxn)
15:56:48 [2018-05-04 15:56:48,482] INFO Client session timed out, have not heard from server in 7583ms for sessionid 0x1000001141b0000, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
15:56:50 [2018-05-04 15:56:50,254] INFO Opening socket connection to server ip-10-0-1-117.eu-west-1.compute.internal/10.0.1.117:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
15:56:56 [2018-05-04 15:56:56,257] WARN Client session timed out, have not heard from server in 7674ms for sessionid 0x1000001141b0000 (org.apache.zookeeper.ClientCnxn)
15:56:56 [2018-05-04 15:56:56,258] INFO Client session timed out, have not heard from server in 7674ms for sessionid 0x1000001141b0000, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
15:56:57 [2018-05-04 15:56:57,591] INFO Opening socket connection to server ip-10-0-1-117.eu-west-1.compute.internal/10.0.1.117:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
15:57:03 [2018-05-04 15:57:03,596] WARN Client session timed out, have not heard from server in 7238ms for sessionid 0x1000001141b0000 (org.apache.zookeeper.ClientCnxn)
15:57:03 [2018-05-04 15:57:03,596] INFO Client session timed out, have not heard from server in 7238ms for sessionid 0x1000001141b0000, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
15:57:04 [2018-05-04 15:57:04,890] INFO Opening socket connection to server ip-10-0-1-117.eu-west-1.compute.internal/10.0.1.117:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
15:57:10 [2018-05-04 15:57:10,894] WARN Client session timed out, have not heard from server in 7198ms for sessionid 0x1000001141b0000 (org.apache.zookeeper.ClientCnxn)
15:57:10 [2018-05-04 15:57:10,894] INFO Client session timed out, have not heard from server in 7198ms for sessionid 0x1000001141b0000, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
15:57:12 [2018-05-04 15:57:12,388] INFO Opening socket connection to server ip-10-0-1-117.eu-west-1.compute.internal/10.0.1.117:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
15:57:18 [2018-05-04 15:57:18,392] WARN Client session timed out, have not heard from server in 7398ms for sessionid 0x1000001141b0000 (org.apache.zookeeper.ClientCnxn)
15:57:18 [2018-05-04 15:57:18,392] INFO Client session timed out, have not heard from server in 7398ms for sessionid 0x1000001141b0000, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
15:57:20 [2018-05-04 15:57:20,162] INFO Opening socket connection to server ip-10-0-1-117.eu-west-1.compute.internal/10.0.1.117:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
15:57:24 [2018-05-04 15:57:24,254] WARN Session 0x1000001141b0000 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
15:57:24 java.net.NoRouteToHostException: Host is unreachable
15:57:24 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
15:57:24 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
15:57:24 at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
15:57:24 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
15:57:25 [2018-05-04 15:57:25,482] INFO Opening socket connection to server ip-10-0-1-117.eu-west-1.compute.internal/10.0.1.117:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
15:57:50 [2018-05-04 15:57:50,783] INFO Opening socket connection to server ip-10-0-1-117.eu-west-1.compute.internal/10.0.1.117:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
15:57:51 [2018-05-04 15:57:51,902] WARN Session 0x1000001141b0000 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
15:57:51 java.net.NoRouteToHostException: Host is unreachable
15:57:51 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
15:57:51 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
15:57:51 at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
15:57:51 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
15:57:52 [2018-05-04 15:57:52,203] INFO Terminating process due to signal SIGTERM (kafka.Kafka$)
15:57:52 [2018-05-04 15:57:52,205] INFO [KafkaServer id=1001] shutting down (kafka.server.KafkaServer)
15:57:52 [2018-05-04 15:57:52,207] INFO [KafkaServer id=1001] Starting controlled shutdown (kafka.server.KafkaServer)
15:57:53 [2018-05-04 15:57:53,962] INFO Opening socket connection to server ip-10-0-1-117.eu-west-1.compute.internal/10.0.1.117:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
15:57:54 [2018-05-04 15:57:54,974] WARN Session 0x1000001141b0000 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
15:57:54 java.net.NoRouteToHostException: Host is unreachable
15:57:54 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
15:57:54 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
15:57:54 at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
15:57:54 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
15:57:55 [2018-05-04 15:57:55,076] INFO [ZooKeeperClient] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
15:57:56 [2018-05-04 15:57:56,978] INFO Opening socket connection to server ip-10-0-1-117.eu-west-1.compute.internal/10.0.1.117:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
15:57:58 [2018-05-04 15:57:58,046] WARN Session 0x1000001141b0000 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)