在 Window 中无法将 csv 文件从 logstash 映射到 kibana

Question

我正在尝试使用 logstash 将 csv 文件中的数据提供给弹性搜索。我的 logsatsh 配置文件如下所示：

input {
file {
path => "D:\Log Anlyser\data\cars.csv"
start_position => "beginning"
sincedb_path => "NUL"   
}
}
filter {
csv {
separator => ","
columns => [ "maker", "model", "mileage", "manufacture_year", "engine_displacement", "engine_power", "body_type", "color_slug", "stk_year", "transmission", "door_count", "seat_count", "fuel_type", "date_created", "date_last_seen", "price_eur" ]
}
mutate {convert => ["milage", "integer"] }
mutate {convert => ["price_eur", "float"] }
mutate {convert => ["engine_power", "integer"] }
mutate {convert => ["door_count", "integer"] }
mutate {convert => ["seat_count", "integer"] }  
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => ["cars-%{+YYYY.MM.dd}"]
}
}

在 window 中为 logstash 触发此命令时：logstash -f cars.conf 我得到这个：-

Sending Logstash logs to D:/Log_Anlyser/logstash/logs which is now configured via log4j2.properties
[2019-02-26T12:05:51,690][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-02-26T12:05:51,721][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.6.1"}
[2019-02-26T12:05:57,133][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-02-26T12:05:57,510][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-02-26T12:05:57,664][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-02-26T12:05:57,711][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>5}
[2019-02-26T12:05:57,742][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-02-26T12:05:57,758][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2019-02-26T12:05:57,852][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2019-02-26T12:05:58,179][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x274079d5 run>"}
[2019-02-26T12:05:58,226][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-02-26T12:05:58,226][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2019-02-26T12:05:58,547][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

现在连接到 kibana(localhost:5601) 时我无法映射数据。 出现此错误：-

Unable to fetch mapping. Do you have indices matching the pattern? Can you please help.

Answer 1

可能有几个原因 - 也许数据根本没有到达 ES。您可以通过 运行

验证索引是否存在来检查

GET es-url:9200/_cat/indices/cars*

如果存在索引，那么您应该能够在 Kibana 中创建索引模式。

如果索引丢失，那么要么是 Logstash 没有读取输入文件，要么是 elasticsearch 不可访问。需要检查logstash日志，确保数据到达ES。

Answer 2

我明白了。错误是很愚蠢的。 CSV 文件的路径错误。较早的路径是 path => "D:\Log Anlyser\data\cars.csv"。当前路径是`

path => "D:/Log_Anlyser/data/cars.csv"

它会起作用