входной плагин logstash s3 не читает файлы из s3 - PullRequest
Новая
       12

входной плагин logstash s3 не читает файлы из s3

0 голосов
/ 25 марта 2019

Я пытаюсь читать журналы из внутреннего хранилища S3 (не AWS), используя logstash.Ниже приведен конфиг.Но когда я пытаюсь это всегда дает никаких файлов, найденных в ведре.Через проводник я мог видеть файлы.Может ли кто-нибудь помочь мне в этом.Я попробовал оба файла .log и .txt в хранилище s3 для целей тестирования.Ничего не работает. 

input {
   s3 {
       access_key_id => "xxxxxxxxxxxxxxxxxxxxxx"
       secret_access_key => "xxxxxxxxxxxxxxxxxxxxx"
       endpoint=>"http://example.com:9020"
       bucket => "samplelogs"
       temporary_directory=>"C:/xxx/ELK/logstash-6.6.1"
       prefix=>"/"
       add_field => { source => gzfiles }
       type => "s3"

   }
}

Это журналы

Отправка журналов Logstash в C: /XXXX/logstash-6.3.2/logs, который теперь настраивается через log4j2.properties 

[2019-03-18T15:58:00,879][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-03-18T15:58:01,311][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.3.2"}
[2019-03-18T15:58:25,213][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-03-18T15:58:25,494][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2019-03-18T15:58:25,509][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2019-03-18T15:58:25,650][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-03-18T15:58:25,681][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-03-18T15:58:25,681][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2019-03-18T15:58:25,713][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-03-18T15:58:25,728][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2019-03-18T15:58:25,744][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2019-03-18T15:58:25,759][INFO ][logstash.inputs.s3 ] Registering s3 input {:bucket=>"gvmslogs", :region=>"us-east-1"}
[2019-03-18T15:58:26,041][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x6ac2a093 run>"}
[2019-03-18T15:58:26,103][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2019-03-18T15:58:26,338][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-03-18T15:58:28,400][INFO ][logstash.inputs.s3 ] S3 input: No files found in bucket {:prefix=>"/"}
[2019-03-18T15:59:27,463][INFO ][logstash.inputs.s3 ] S3 input: No files found in bucket {:prefix=>"/"}
...