stdout 将 404 放入 logstash
stdout put 404 in logstash
我是 elk stack 的新手，我正在尝试做一个非常基本的实验：使用 PUT 请求向 logstash stdout 发送消息，基于此 repo：link
logstash的端口是9600，我用postman发送了一个PUT请求。它returns 404
我的logstash.conf很简单
input {
http {
}
}
output {
stdout {
}
}
至于docker-compose文件中的设置，这里是：
logstash:
build:
context: logstash/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- type: bind
source: ./logstash/config/logstash.yml
target: /usr/share/logstash/config/logstash.yml
read_only: true
- type: bind
source: ./logstash/pipeline
target: /usr/share/logstash/pipeline
read_only: true
ports:
- "5044:5044"
- "5000:5000/tcp"
- "5000:5000/udp"
- "9600:9600"
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
networks:
- elk
depends_on:
- elasticsearch
GET 请求有效，结果如下：
{
"host": "b32085c40331",
"version": "7.10.2",
"http_address": "0.0.0.0:9600",
"id": "0079f53f-1d2e-4278-85eb-0817fa95506c",
"name": "b32085c40331",
"ephemeral_id": "d0c18df3-9a0b-48c9-abb4-9e41543ed7ac",
"status": "green",
"snapshot": false,
"pipeline": {
"workers": 4,
"batch_size": 125,
"batch_delay": 50
},
"monitoring": {
"hosts": [
"http://elasticsearch:9200"
],
"username": "elastic"
},
"build_date": "2021-01-13T02:43:06Z",
"build_sha": "7cebafee7a073fa9d58c97de074064a540d6c317",
"build_snapshot": false
}
关于 logstash，docker-compose logs logstash，我得到一个大日志，我什至不知道从哪里开始：
logstash_1 | Using bundled JDK: /usr/share/logstash/jdk
logstash_1 | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash_1 | Using bundled JDK: /usr/share/logstash/jdk
logstash_1 | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash_1 | WARNING: An illegal reflective access operation has occurred
logstash_1 | WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-1/jruby5118775578707886457jopenssl.jar) to field java.security.MessageDigest.provider
logstash_1 | WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
logstash_1 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
logstash_1 | WARNING: All illegal access operations will be denied in a future release
logstash_1 | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash_1 | [2021-01-29T12:00:35,199][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.10.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
logstash_1 | [2021-01-29T12:00:35,412][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
logstash_1 | [2021-01-29T12:00:35,440][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
logstash_1 | [2021-01-29T12:00:37,687][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"0079f53f-1d2e-4278-85eb-0817fa95506c", :path=>"/usr/share/logstash/data/uuid"}
logstash_1 | [2021-01-29T12:00:38,657][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
logstash_1 | Please configure Metricbeat to monitor Logstash. Documentation can be found at:
logstash_1 | https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
logstash_1 | [2021-01-29T12:00:42,951][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1 | [2021-01-29T12:00:46,669][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@elasticsearch:9200/]}}
logstash_1 | [2021-01-29T12:00:50,290][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@elasticsearch:9200/"}
logstash_1 | [2021-01-29T12:00:50,515][INFO ][logstash.licensechecker.licensereader] ES Output version determined {:es_version=>7}
logstash_1 | [2021-01-29T12:00:50,518][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
logstash_1 | [2021-01-29T12:00:50,694][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
logstash_1 | [2021-01-29T12:00:50,695][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
logstash_1 | [2021-01-29T12:00:53,243][INFO ][org.reflections.Reflections] Reflections took 606 ms to scan 1 urls, producing 23 keys and 47 values
logstash_1 | [2021-01-29T12:00:54,045][WARN ][deprecation.logstash.outputs.elasticsearchmonitoring] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1 | [2021-01-29T12:00:54,339][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@elasticsearch:9200/]}}
logstash_1 | [2021-01-29T12:00:54,417][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@elasticsearch:9200/"}
logstash_1 | [2021-01-29T12:00:54,500][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] ES Output version determined {:es_version=>7}
logstash_1 | [2021-01-29T12:00:54,500][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
logstash_1 | [2021-01-29T12:00:54,627][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://elasticsearch:9200"]}
logstash_1 | [2021-01-29T12:00:54,691][WARN ][logstash.javapipeline ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
logstash_1 | [2021-01-29T12:00:54,953][INFO ][logstash.javapipeline ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x37941be run>"}
logstash_1 | [2021-01-29T12:00:55,984][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x3e7f065e run>"}
logstash_1 | [2021-01-29T12:01:00,012][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>5.05}
logstash_1 | [2021-01-29T12:01:00,013][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>4.03}
logstash_1 | [2021-01-29T12:01:00,142][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
logstash_1 | [2021-01-29T12:01:01,027][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"}
logstash_1 | [2021-01-29T12:01:01,209][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
logstash_1 | [2021-01-29T12:01:01,245][INFO ][logstash.inputs.http ][main][2d26a22d7786b5d1d6a62684242754061f0e7699167308954d8cf88e52c80903] Starting http input listener {:address=>"0.0.0.0:8080", :ssl=>"false"}
logstash_1 | [2021-01-29T12:01:01,217][INFO ][logstash.inputs.tcp ][main][6ca97606e772405a9e65bc09f9b369d784557cb3e3fea379b981c5d16a9573f1] Starting tcp input listener {:address=>"0.0.0.0:5000", :ssl_enable=>"false"}
logstash_1 | [2021-01-29T12:01:01,306][INFO ][org.logstash.beats.Server][main][d704d487716580c50daa3a9bb4e99ad2bfa9542e31e8b0b06a9e0ea687e6f15a] Starting server on port: 5044
logstash_1 | [2021-01-29T12:01:01,340][INFO ][logstash.agent ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}
logstash_1 | [2021-01-29T12:01:02,200][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
如何解决这个问题？
端口 9600 是 Logstash API 的端口，用于监控 logstash，而不是 http 输入的端口。
如果您想使用 http 输入并且由于您没有在配置中指定端口，您应该使用端口 8080，即 default port这个输入。
您还需要在 docker 配置中公开此端口。
我是 elk stack 的新手，我正在尝试做一个非常基本的实验：使用 PUT 请求向 logstash stdout 发送消息，基于此 repo：link
logstash的端口是9600，我用postman发送了一个PUT请求。它returns 404
我的logstash.conf很简单
input {
http {
}
}
output {
stdout {
}
}
至于docker-compose文件中的设置，这里是：
logstash:
build:
context: logstash/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- type: bind
source: ./logstash/config/logstash.yml
target: /usr/share/logstash/config/logstash.yml
read_only: true
- type: bind
source: ./logstash/pipeline
target: /usr/share/logstash/pipeline
read_only: true
ports:
- "5044:5044"
- "5000:5000/tcp"
- "5000:5000/udp"
- "9600:9600"
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
networks:
- elk
depends_on:
- elasticsearch
GET 请求有效，结果如下：
{
"host": "b32085c40331",
"version": "7.10.2",
"http_address": "0.0.0.0:9600",
"id": "0079f53f-1d2e-4278-85eb-0817fa95506c",
"name": "b32085c40331",
"ephemeral_id": "d0c18df3-9a0b-48c9-abb4-9e41543ed7ac",
"status": "green",
"snapshot": false,
"pipeline": {
"workers": 4,
"batch_size": 125,
"batch_delay": 50
},
"monitoring": {
"hosts": [
"http://elasticsearch:9200"
],
"username": "elastic"
},
"build_date": "2021-01-13T02:43:06Z",
"build_sha": "7cebafee7a073fa9d58c97de074064a540d6c317",
"build_snapshot": false
}
关于 logstash，docker-compose logs logstash，我得到一个大日志，我什至不知道从哪里开始：
logstash_1 | Using bundled JDK: /usr/share/logstash/jdk
logstash_1 | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash_1 | Using bundled JDK: /usr/share/logstash/jdk
logstash_1 | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash_1 | WARNING: An illegal reflective access operation has occurred
logstash_1 | WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-1/jruby5118775578707886457jopenssl.jar) to field java.security.MessageDigest.provider
logstash_1 | WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
logstash_1 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
logstash_1 | WARNING: All illegal access operations will be denied in a future release
logstash_1 | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash_1 | [2021-01-29T12:00:35,199][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.10.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
logstash_1 | [2021-01-29T12:00:35,412][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
logstash_1 | [2021-01-29T12:00:35,440][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
logstash_1 | [2021-01-29T12:00:37,687][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"0079f53f-1d2e-4278-85eb-0817fa95506c", :path=>"/usr/share/logstash/data/uuid"}
logstash_1 | [2021-01-29T12:00:38,657][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
logstash_1 | Please configure Metricbeat to monitor Logstash. Documentation can be found at:
logstash_1 | https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
logstash_1 | [2021-01-29T12:00:42,951][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1 | [2021-01-29T12:00:46,669][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@elasticsearch:9200/]}}
logstash_1 | [2021-01-29T12:00:50,290][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@elasticsearch:9200/"}
logstash_1 | [2021-01-29T12:00:50,515][INFO ][logstash.licensechecker.licensereader] ES Output version determined {:es_version=>7}
logstash_1 | [2021-01-29T12:00:50,518][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
logstash_1 | [2021-01-29T12:00:50,694][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
logstash_1 | [2021-01-29T12:00:50,695][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
logstash_1 | [2021-01-29T12:00:53,243][INFO ][org.reflections.Reflections] Reflections took 606 ms to scan 1 urls, producing 23 keys and 47 values
logstash_1 | [2021-01-29T12:00:54,045][WARN ][deprecation.logstash.outputs.elasticsearchmonitoring] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1 | [2021-01-29T12:00:54,339][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@elasticsearch:9200/]}}
logstash_1 | [2021-01-29T12:00:54,417][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@elasticsearch:9200/"}
logstash_1 | [2021-01-29T12:00:54,500][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] ES Output version determined {:es_version=>7}
logstash_1 | [2021-01-29T12:00:54,500][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
logstash_1 | [2021-01-29T12:00:54,627][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://elasticsearch:9200"]}
logstash_1 | [2021-01-29T12:00:54,691][WARN ][logstash.javapipeline ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
logstash_1 | [2021-01-29T12:00:54,953][INFO ][logstash.javapipeline ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x37941be run>"}
logstash_1 | [2021-01-29T12:00:55,984][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x3e7f065e run>"}
logstash_1 | [2021-01-29T12:01:00,012][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>5.05}
logstash_1 | [2021-01-29T12:01:00,013][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>4.03}
logstash_1 | [2021-01-29T12:01:00,142][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
logstash_1 | [2021-01-29T12:01:01,027][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"}
logstash_1 | [2021-01-29T12:01:01,209][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
logstash_1 | [2021-01-29T12:01:01,245][INFO ][logstash.inputs.http ][main][2d26a22d7786b5d1d6a62684242754061f0e7699167308954d8cf88e52c80903] Starting http input listener {:address=>"0.0.0.0:8080", :ssl=>"false"}
logstash_1 | [2021-01-29T12:01:01,217][INFO ][logstash.inputs.tcp ][main][6ca97606e772405a9e65bc09f9b369d784557cb3e3fea379b981c5d16a9573f1] Starting tcp input listener {:address=>"0.0.0.0:5000", :ssl_enable=>"false"}
logstash_1 | [2021-01-29T12:01:01,306][INFO ][org.logstash.beats.Server][main][d704d487716580c50daa3a9bb4e99ad2bfa9542e31e8b0b06a9e0ea687e6f15a] Starting server on port: 5044
logstash_1 | [2021-01-29T12:01:01,340][INFO ][logstash.agent ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}
logstash_1 | [2021-01-29T12:01:02,200][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
如何解决这个问题？
端口 9600 是 Logstash API 的端口，用于监控 logstash，而不是 http 输入的端口。
如果您想使用 http 输入并且由于您没有在配置中指定端口，您应该使用端口 8080，即 default port这个输入。
您还需要在 docker 配置中公开此端口。