如何为 Swisscom elasticsearch public 云配置 Kibana (CloudFoundry)
How to configure Kibana for Swisscom elasticsearch public cloud (CloudFoundry)
注意：此问题针对Swisscom提供的Elasticsearch服务
问题: (a.k.a: tl;dr)
需要什么配置才能让官方 Kibana docker 容器连接到 Swisscom Elasticsearch 服务？
背景:
直到大约一年前 Swisscom public cloud offered a full ELK stack (Elasticsearch, Logstash, Kibana) in a single service offering. When this service was discontinued, Swisscom replaced it by just offering the Elasticsearch service and asked clients to setup their own Kibana and Logstash solutions via provided CloudFoundry build_packs (Kibana, Logstash). The migration recommendation was discussed here: https://ict.swisscom.ch/2018/04/building-the-elk-stack-on-our-new-elasticsearch/
最近，在 Swisscom 基于 CloudFoundry 的 PaaS 产品上运行应用程序的底层 OS（称为 "stack"）已经升级。前面提到的 build_packs 现在已经过时，并且已被 Swisscom 宣布为 弃用 。现在的建议是转向通用 Docker container provided by Elastic as discussed here: https://github.com/swisscom/kibana-buildpack/issues/3
我试过的：
CloudFoundry 通常与 Docker 容器配合良好，整个过程应该像向 docker 容器提供一些有效配置一样简单明了。我当前的 Kibana manifest.yml 看起来像这样，但 Kibana 应用程序最终无法连接：
---
applications:
- name: kibana-test-example
docker:
image: docker.elastic.co/kibana/kibana:6.1.4
memory: 4G
disk_quota: 5G
services:
- elasticsearch-test-service
env:
SERVER_NAME: kibana-test
ELASTICSEARCH_URL: https://abcdefghijk.elasticsearch.lyra-836.appcloud.swisscom.com
ELASTICSEARCH_USERNAME: username_provided_by_elasticsearch_service
ELASTICSEARCH_PASSWORD: password_provided_by_elasticsearch_service
XPACK_MONITORING_ENABLED: true
附加信息：
Swisscom 提供的 Elasticsearch Service 当前运行版本 6.1.3。据我所知，它已安装 x-pack。
你遇到了什么错误？
我试了一下配置，发现了不同的错误，其中大部分似乎与针对 Elasticsearch 服务的身份验证失败有关。
这是一些示例性的初始日志输出（不过说真的，您需要一个 运行 Kibana 才能读取它...）
2019-05-10T08:08:34.43+0200 [CELL/0] OUT Cell eda692ed-f4c3-4a5e-86aa-c0d1641b029f successfully created container for instance 385e5b7f-1570-46cd-532a-c5b4
2019-05-10T08:08:48.60+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:08:48Z","tags":["info","optimize"],"pid":6,"message":"Optimizing and caching bundles for graph, monitoring, apm, kibana, stateSessionStorageRedirect, timelion, login, logout, dashboardViewer and status_page. This may take a few minutes"}
2019-05-10T08:15:07.68+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["info","optimize"],"pid":6,"message":"Optimization of bundles for graph, monitoring, apm, kibana, stateSessionStorageRedirect, timelion, login, logout, dashboardViewer and status_page complete in 379.08 seconds"}
2019-05-10T08:15:07.77+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:kibana@6.1.4","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
2019-05-10T08:15:07.82+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:elasticsearch@6.1.4","info"],"pid":6,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
2019-05-10T08:15:07.86+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:xpack_main@6.1.4","info"],"pid":6,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
2019-05-10T08:15:07.86+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:graph@6.1.4","info"],"pid":6,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
2019-05-10T08:15:07.88+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:monitoring@6.1.4","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
2019-05-10T08:15:07.89+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:xpack_main@6.1.4","error"],"pid":6,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
2019-05-10T08:15:07.89+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:graph@6.1.4","error"],"pid":6,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
2019-05-10T08:15:07.89+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:elasticsearch@6.1.4","error"],"pid":6,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
2019-05-10T08:15:11.39+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:11Z","tags":["reporting","warning"],"pid":6,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
2019-05-10T08:15:11.39+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:11Z","tags":["status","plugin:reporting@6.1.4","error"],"pid":6,"state":"red","message":"Status changed from uninitialized to red - Authentication Exception","prevState":"uninitialized","prevMsg":"uninitialized"}
实际相关的错误信息似乎是这样的：
2019-05-10T08:15:11.66+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:11Z","tags":["license","warning","xpack"],"pid":6,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_exception] unable to authenticate user [ABCDEFGHIJKLMNOPQRST] for REST request [/_xpack], with { header={ WWW-Authenticate=\"Basic realm=\\"security\\" charset=\\"UTF-8\\"\" } } :: {\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\"error\\":{\\"root_cause\\":[{\\"type\\":\\"security_exception\\",\\"reason\\":\\"unable to authenticate user [ABCDEFGHIJKLMNOPQRST] for REST request [/_xpack]\\",\\"header\\":{\\"WWW-Authenticate\\":\\"Basic realm=\\\\"security\\\\" charset=\\\\"UTF-8\\\\"\\"}}],\\"type\\":\\"security_exception\\",\\"reason\\":\\"unable to authenticate user [ABCDEFGHIJKLMNOPQRST] for REST request [/_xpack]\\",\\"header\\":{\\"WWW-Authenticate\\":\\"Basic realm=\\\\"security\\\\" charset=\\\\"UTF-8\\\\"\\"}},\\"status\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\"security\\" charset=\\"UTF-8\\"\"}"}
我尝试按照其他地方的建议设置 XPACK_SECURITY_ENABLED: false 以及设置实际的 SERVER_HOST，这似乎让事情变得更糟。
我非常感谢有人使用现有 Kibana docker 图像连接到 Swisscom 提供的 Elasticsearch 服务的工作示例。
会不会是你混淆了用户名和密码？当我检查我的 service-key 密码在用户名之前时，这可能会导致您这边出现 copy-paste 错误：
cf service-key myece mykey|grep kibana_system
"kibana_system_password": "aKvOpMVrXGCJ4PJht",
"kibana_system_username": "aksTxVNyLU4JWiQOE6V",
我尝试使用您的 manifest.yml 推送 Kibana，它在我的情况下非常有效。
Swisscom 还更新了有关如何通过 Docker 使用 Kibana 和 Logstash 的文档：
https://docs.developer.swisscom.com/service-offerings/kibana-docker.html
https://docs.developer.swisscom.com/service-offerings/logstash-docker.html
注意：此问题针对Swisscom提供的Elasticsearch服务
问题: (a.k.a: tl;dr)
需要什么配置才能让官方 Kibana docker 容器连接到 Swisscom Elasticsearch 服务？
背景:
直到大约一年前 Swisscom public cloud offered a full ELK stack (Elasticsearch, Logstash, Kibana) in a single service offering. When this service was discontinued, Swisscom replaced it by just offering the Elasticsearch service and asked clients to setup their own Kibana and Logstash solutions via provided CloudFoundry build_packs (Kibana, Logstash). The migration recommendation was discussed here: https://ict.swisscom.ch/2018/04/building-the-elk-stack-on-our-new-elasticsearch/
最近，在 Swisscom 基于 CloudFoundry 的 PaaS 产品上运行应用程序的底层 OS（称为 "stack"）已经升级。前面提到的 build_packs 现在已经过时，并且已被 Swisscom 宣布为 弃用 。现在的建议是转向通用 Docker container provided by Elastic as discussed here: https://github.com/swisscom/kibana-buildpack/issues/3
我试过的：
CloudFoundry 通常与 Docker 容器配合良好，整个过程应该像向 docker 容器提供一些有效配置一样简单明了。我当前的 Kibana manifest.yml 看起来像这样，但 Kibana 应用程序最终无法连接：
---
applications:
- name: kibana-test-example
docker:
image: docker.elastic.co/kibana/kibana:6.1.4
memory: 4G
disk_quota: 5G
services:
- elasticsearch-test-service
env:
SERVER_NAME: kibana-test
ELASTICSEARCH_URL: https://abcdefghijk.elasticsearch.lyra-836.appcloud.swisscom.com
ELASTICSEARCH_USERNAME: username_provided_by_elasticsearch_service
ELASTICSEARCH_PASSWORD: password_provided_by_elasticsearch_service
XPACK_MONITORING_ENABLED: true
附加信息：
Swisscom 提供的 Elasticsearch Service 当前运行版本 6.1.3。据我所知，它已安装 x-pack。
你遇到了什么错误？
我试了一下配置，发现了不同的错误，其中大部分似乎与针对 Elasticsearch 服务的身份验证失败有关。
这是一些示例性的初始日志输出（不过说真的，您需要一个 运行 Kibana 才能读取它...）
2019-05-10T08:08:34.43+0200 [CELL/0] OUT Cell eda692ed-f4c3-4a5e-86aa-c0d1641b029f successfully created container for instance 385e5b7f-1570-46cd-532a-c5b4
2019-05-10T08:08:48.60+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:08:48Z","tags":["info","optimize"],"pid":6,"message":"Optimizing and caching bundles for graph, monitoring, apm, kibana, stateSessionStorageRedirect, timelion, login, logout, dashboardViewer and status_page. This may take a few minutes"}
2019-05-10T08:15:07.68+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["info","optimize"],"pid":6,"message":"Optimization of bundles for graph, monitoring, apm, kibana, stateSessionStorageRedirect, timelion, login, logout, dashboardViewer and status_page complete in 379.08 seconds"}
2019-05-10T08:15:07.77+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:kibana@6.1.4","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
2019-05-10T08:15:07.82+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:elasticsearch@6.1.4","info"],"pid":6,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
2019-05-10T08:15:07.86+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:xpack_main@6.1.4","info"],"pid":6,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
2019-05-10T08:15:07.86+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:graph@6.1.4","info"],"pid":6,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
2019-05-10T08:15:07.88+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:monitoring@6.1.4","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
2019-05-10T08:15:07.89+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:xpack_main@6.1.4","error"],"pid":6,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
2019-05-10T08:15:07.89+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:graph@6.1.4","error"],"pid":6,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
2019-05-10T08:15:07.89+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:elasticsearch@6.1.4","error"],"pid":6,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
2019-05-10T08:15:11.39+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:11Z","tags":["reporting","warning"],"pid":6,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
2019-05-10T08:15:11.39+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:11Z","tags":["status","plugin:reporting@6.1.4","error"],"pid":6,"state":"red","message":"Status changed from uninitialized to red - Authentication Exception","prevState":"uninitialized","prevMsg":"uninitialized"}
实际相关的错误信息似乎是这样的：
2019-05-10T08:15:11.66+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:11Z","tags":["license","warning","xpack"],"pid":6,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_exception] unable to authenticate user [ABCDEFGHIJKLMNOPQRST] for REST request [/_xpack], with { header={ WWW-Authenticate=\"Basic realm=\\"security\\" charset=\\"UTF-8\\"\" } } :: {\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\"error\\":{\\"root_cause\\":[{\\"type\\":\\"security_exception\\",\\"reason\\":\\"unable to authenticate user [ABCDEFGHIJKLMNOPQRST] for REST request [/_xpack]\\",\\"header\\":{\\"WWW-Authenticate\\":\\"Basic realm=\\\\"security\\\\" charset=\\\\"UTF-8\\\\"\\"}}],\\"type\\":\\"security_exception\\",\\"reason\\":\\"unable to authenticate user [ABCDEFGHIJKLMNOPQRST] for REST request [/_xpack]\\",\\"header\\":{\\"WWW-Authenticate\\":\\"Basic realm=\\\\"security\\\\" charset=\\\\"UTF-8\\\\"\\"}},\\"status\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\"security\\" charset=\\"UTF-8\\"\"}"}
我尝试按照其他地方的建议设置 XPACK_SECURITY_ENABLED: false 以及设置实际的 SERVER_HOST，这似乎让事情变得更糟。
我非常感谢有人使用现有 Kibana docker 图像连接到 Swisscom 提供的 Elasticsearch 服务的工作示例。
会不会是你混淆了用户名和密码？当我检查我的 service-key 密码在用户名之前时，这可能会导致您这边出现 copy-paste 错误：
cf service-key myece mykey|grep kibana_system
"kibana_system_password": "aKvOpMVrXGCJ4PJht",
"kibana_system_username": "aksTxVNyLU4JWiQOE6V",
我尝试使用您的 manifest.yml 推送 Kibana，它在我的情况下非常有效。
Swisscom 还更新了有关如何通过 Docker 使用 Kibana 和 Logstash 的文档：
https://docs.developer.swisscom.com/service-offerings/kibana-docker.html https://docs.developer.swisscom.com/service-offerings/logstash-docker.html