Logsta sh не вставляет записи в ElasticSearch - PullRequest
Новая
       6

Logsta sh не вставляет записи в ElasticSearch

0 голосов
/ 17 апреля 2020

Я пытаюсь вставить простые значения CSV в ES. это не происходит через. Файл conf, который я использую, выглядит следующим образом: 

input {
  file {
    path => "D:/ELK/Logs/rahul.csv"
    start_position => "beginning"
    sincedb_path => "NULL"
  }
}

filter {
  csv {
    columns => [ "record_id", "duration", "src_bytes", "dest_bytes" ]
  }
}

output {
  elasticsearch { 
    hosts => "http://localhost:9200"
    index => "rahul"
  }
  stdout {}
}

Я проверил журнал ошибок в папке Logstash \ logs. следующее есть 

[2020-04-17T17:07:34,598][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-04-17T17:07:34,662][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.6.2"}
[2020-04-17T17:07:35,419][INFO ][org.reflections.Reflections] Reflections took 25 ms to scan 1 urls, producing 20 keys and 40 values 
[2020-04-17T17:07:36,217][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2020-04-17T17:07:36,328][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-04-17T17:07:36,359][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-04-17T17:07:36,362][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-04-17T17:07:36,393][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2020-04-17T17:07:36,423][INFO ][logstash.outputs.elasticsearch][main] Using default mapping template
[2020-04-17T17:07:36,443][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[2020-04-17T17:07:36,447][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>6, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>750, "pipeline.sources"=>["D:/ELK/Logs/rahul.conf"], :thread=>"#<Thread:0x5e5be359 run>"}
[2020-04-17T17:07:36,467][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-04-17T17:07:37,167][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-04-17T17:07:37,188][INFO ][filewatch.observingtail  ][main] START, creating Discoverer, Watch with file and sincedb collections
[2020-04-17T17:07:37,205][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-04-17T17:07:37,395][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

1 Ответ

0 голосов
/ 18 апреля 2020

разрешается добавлением sincedb_path => "NULL" к входному плагину, например: 

input {
  file {
    path => "D:/ELK/Logs/rahul.csv"
    start_position => "beginning"
    sincedb_path => "NULL" 
  }
}
...