Я сослался на это Я понял, что он устарел.
У меня есть данные, в которых одно поле является первичным ключом. Мне нужно загрузить через log sta sh. Для этого я пытаюсь создать сопоставление, как указано в потоке.
{
"mappings" : {
"doc":{
"_id" : {
"path":"mini_system_key"
}
},
"properties" : {
"mini_system_key" : { "type" : "keyword", "index":"true", "store":"true" }
}
}
}
Это вызывает следующую ошибку.
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [doc : {_id={path=mini_system_key}}]"
}
],
"type": "mapper_parsing_exception",
"reason": "Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters: [doc : {_id={path=mini_system_key}}]",
"caused_by": {
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [doc : {_id={path=mini_system_key}}]"
}
},
"status": 400
}
Как мне этого добиться? Я хочу иметь настраиваемое поле в качестве поля идентификатора?
Журнал sh:
input {
file {
path => "/Users/gibbs/Documents/search/mini_system_data.csv"
start_position => beginning
}
}
filter {
csv {
columns => [
"mini_system_key"
]
separator => ","
}
mutate {
remove_field => ["path", "host"]
}
}
output {
stdout
{
codec => rubydebug
}
elasticsearch {
action => "index"
hosts => ["127.0.0.1:9200"]
index => "mini_system"
document_id => "%{mini_system_key}"
}
}
Когда я пытаюсь это сделать, данные не загружаются.
Он зависает, как показано ниже.
./logstash -f ../config/mini_system_schema.conf
Sending Logstash logs to /Users/gibbs/Documents/logstash-7.6.2/logs which is now configured via log4j2.properties
[2020-05-09T20:41:47,011][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-05-09T20:41:47,124][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.6.2"}
[2020-05-09T20:41:48,846][INFO ][org.reflections.Reflections] Reflections took 36 ms to scan 1 urls, producing 20 keys and 40 values
[2020-05-09T20:41:50,657][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://127.0.0.1:9200/]}}
[2020-05-09T20:41:50,902][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://127.0.0.1:9200/"}
[2020-05-09T20:41:50,964][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-05-09T20:41:50,970][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-05-09T20:41:51,004][WARN ][logstash.outputs.elasticsearch][main] DEPRECATION WARNING: Connecting to an OSS distribution of Elasticsearch using the default distribution of Logstash will stop working in Logstash 8.0.0. Please upgrade to the default distribution of Elasticsearch, or use the OSS distribution of Logstash {:url=>"http://127.0.0.1:9200/"}
[2020-05-09T20:41:51,036][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1:9200"]}
[2020-05-09T20:41:51,122][INFO ][logstash.outputs.elasticsearch][main] Using default mapping template
[2020-05-09T20:41:51,191][INFO ][logstash.outputs.elasticsearch][main] Index Lifecycle Management is set to 'auto', but will be disabled - Index Lifecycle management is not installed on your Elasticsearch cluster
[2020-05-09T20:41:51,193][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2020-05-09T20:41:51,195][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-05-09T20:41:51,210][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/Users/gibbs/Documents/logstash-7.6.2/config/mini_system_schema.conf"], :thread=>"#<Thread:0x3574edd3 run>"}
[2020-05-09T20:41:52,351][INFO ][logstash.inputs.file ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/Users/gibbs/Documents/logstash-7.6.2/data/plugins/inputs/file/.sincedb_70546b8e568daf46c8d1fca113147a9b", :path=>["/Users/gibbs/Documents/search/mini_system_data.csv"]}
[2020-05-09T20:41:52,465][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-05-09T20:41:52,578][INFO ][filewatch.observingtail ][main] START, creating Discoverer, Watch with file and sincedb collections
[2020-05-09T20:41:52,584][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-05-09T20:41:52,928][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
EDIT:
Данные загружаются нормально из Logsta sh в ES. Но _id не работает.