Я работал над созданием приложения системного журнала, в котором мне нужно было собирать различные типы журналов, сгенерированных с помощью Fluentd, и отправлять их в Cassandra для последующего анализа.Я должен использовать Кафку в качестве буфера или потокового процессора.Для использования Kafka с Fluentd мне нужно настроить Fluentd.
Я установил Fluentd, Cassandra, Kafka и Zookeeper, все они работают без ошибок независимо.Но когда я пытаюсь подключить Kafka к fluentd с помощью файла fluent.conf, я сталкиваюсь с несколькими ошибками.
Мой fluent.conf файл:
<source>
@type syslog
bind 0.0.0.0
port 5140
tag default
protocol_type udp
source_hostname_key source_host
<parse>
message_format auto
</parse>
</source>
<match **>
@type stdout
</match>
<match app.**>
@type kafka_buffered
zookeeper localhost:2181
zookeeper_path root/attackfence/kafka_2.11-2.2.0/zookeeeper
partition_key partition
partition_key_key partition_key
message_key_key message_key
default_topic nil
default_partition_key nil
default_message_key nil
output_data_type json
output_include_tag false
output_include_time false
exclude_topic_key false
exclude_partition_key false
get_kafka_client_log true
</match>
Ошибки, которые я получаюявляются:
2019-05-23 18:22:44 +0530 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `each' for nil:NilClass"
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluent-plugin-kafka-0.9.4/lib/fluent/plugin/out_kafka_buffered.rb:117:in `refresh_client'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluent-plugin-kafka-0.9.4/lib/fluent/plugin/out_kafka_buffered.rb:208:in `start'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/compat/call_super_mixin.rb:42:in `start'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:165:in `block in start'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:154:in `block (2 levels) in lifecycle'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:153:in `each'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:153:in `block in lifecycle'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:140:in `each'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:140:in `lifecycle'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:164:in `start'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/engine.rb:274:in `start'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/engine.rb:219:in `run'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/supervisor.rb:799:in `run_engine'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/supervisor.rb:549:in `block in run_worker'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/supervisor.rb:724:in `main_process'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/supervisor.rb:544:in `run_worker'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/rubies/ruby-2.6.0/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/rubies/ruby-2.6.0/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/bin/fluentd:8:in `<top (required)>'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/bin/fluentd:23:in `load'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/bin/fluentd:23:in `<main>'