EKS - Envoy 动态转发代理 CA 验证错误
EKS - Envoy dynamic forward proxy CA validation error
我正在尝试使用 envoy sidecar-containers 在 eks 中设置一个转发代理，它应该进行 TLS 发起；
IE。特使从主容器获取 http 请求，执行 ssl 加密并使用文件 ca-certificates.crt 验证 ca。
envoy-configuration 基于官方文档(https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/dynamic_forward_proxy_filter#config-http-filters-dynamic-forward-proxy).
特使将 ca 文件 ca-certificates.crt 从 aws secrets manager 挂载到 /etc/cert-volume/ca-certificates.crt，这似乎有效，因为该文件存在于容器中。
ca-certificates.crt 文件似乎也有效，因为来自我本地计算机 的以下命令 运行 给出了 HTTP 302找到（远程 wget 不支持 --ca-certificate 参数）：
wget --ca-certificate=ca-certificates.crt https://git.tech.rz.blub.de
--2021-04-26 11:03:16-- https://git.tech.rz.blub.de/
Loaded CA certificate 'ca-certificates.crt'
Resolving git.tech.rz.blub.de (git.tech.rz.blub.de)... 10.104.85.137, 10.104.84.192, 10.104.86.175
Connecting to git.tech.rz.blub.de (git.tech.rz.blub.de)|10.104.85.137|:443... connected.
HTTP request sent, awaiting response... 302 Found
当我在特使容器上打开 shell 并尝试通过代理发送 http 请求时，我收到 503 服务不可用：
export http_proxy="localhost:10000"
wget git.tech.rz.blub.de
Connecting to localhost:10000 (127.0.0.1:10000)
wget: server returned error: HTTP/1.1 503 Service Unavailable
如特使日志所示，请求已处理并且 DNS 解析成功，但是 ca 验证失败并出现错误 TLS 错误：268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
可以在此处找到日志：https://pastebin.com/pUZZqhSA
完整的特使配置如下所示：
admin:
access_log_path: /dev/stdout
address:
socket_address: { address: 127.0.0.1, port_value: 9901, protocol: TCP }
static_resources:
secrets:
- name: server_cert
tls_certificate:
certificate_chain:
filename: /etc/cert-volume/server.crt
private_key:
filename: /etc/cert-volume/server.key
- name: validation_context
validation_context:
trusted_ca:
filename: /etc/cert-volume/ca-certificates.crt
listeners:
- name: listener_incoming
address:
socket_address: { address: 0.0.0.0, port_value: 8443 }
filter_chains:
filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/config/" }
route: { cluster: local_config }
- match: { prefix: "/basic/" }
route: { cluster: local }
- match: { prefix: "/" }
route: { cluster: local }
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/stdout
log_format:
json_format:
protocol: "%PROTOCOL%"
response_code: "%RESPONSE_CODE%"
response_code_details: "%RESPONSE_CODE_DETAILS%"
connection_termination_details: "%CONNECTION_TERMINATION_DETAILS%"
message: "%LOCAL_REPLY_BODY%"
response_flags: "%RESPONSE_FLAGS%"
local_reply_body: "%LOCAL_REPLY_BODY%"
http_filters:
- name: envoy.filters.http.router
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificate_sds_secret_configs:
- name: server_cert
- name: listener_outgoing
address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: AUTO
stat_prefix: http_to_https
route_config:
name: local_route
virtual_hosts:
- name: app
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: dynamic_forward_proxy_cluster
http_filters:
- name: envoy.filters.http.dynamic_forward_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig
dns_cache_config:
name: dynamic_forward_proxy_cache_config
dns_lookup_family: V4_ONLY
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: local
connect_timeout: 0.25s
type: STATIC
load_assignment:
cluster_name: local
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8080
- name: local_config
connect_timeout: 0.25s
type: STATIC
load_assignment:
cluster_name: local_config
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8081
- name: dynamic_forward_proxy_cluster
connect_timeout: 1s
lb_policy: CLUSTER_PROVIDED
cluster_type:
name: envoy.clusters.dynamic_forward_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig
dns_cache_config:
name: dynamic_forward_proxy_cache_config
dns_lookup_family: V4_ONLY
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
validation_context_sds_secret_config:
name: validation_context
envoy-admin 端点显示，使用了正确的证书：
wget localhost:9901/certs
Connecting to localhost:9901 (127.0.0.1:9901)
saving to 'certs'
certs 100% |*********************************************| 710 0:00:00 ETA
'certs' saved
/ # cat certs
{
"certificates": [
{
"ca_cert": [
{
"path": "/etc/cert-volume/ca-certificates.crt",
"serial_number": "612a55dd000000000002",
"subject_alt_names": [],
"days_until_expiration": "1305",
"valid_from": "2013-11-21T15:40:58Z",
"expiration_time": "2024-11-21T15:50:58Z"
}
],
"cert_chain": []
},
{
"ca_cert": [],
"cert_chain": [
{
"path": "/etc/cert-volume/server.crt",
"serial_number": "74af53b8463a1714",
"subject_alt_names": [
{
"dns": "*.test.comp.blub.de"
}
],
"days_until_expiration": "173",
"valid_from": "2019-10-18T07:10:38Z",
"expiration_time": "2021-10-17T07:10:38Z"
}
]
}
]
}
有没有人知道，这里出了什么问题？
问题已解决。
ca-certificates.crt 缺少必需的 CA。我不知道为什么 wget 没有抛出异常，但是 openssl 抛出异常...
我正在尝试使用 envoy sidecar-containers 在 eks 中设置一个转发代理，它应该进行 TLS 发起； IE。特使从主容器获取 http 请求，执行 ssl 加密并使用文件 ca-certificates.crt 验证 ca。 envoy-configuration 基于官方文档(https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/dynamic_forward_proxy_filter#config-http-filters-dynamic-forward-proxy).
特使将 ca 文件 ca-certificates.crt 从 aws secrets manager 挂载到 /etc/cert-volume/ca-certificates.crt，这似乎有效，因为该文件存在于容器中。
ca-certificates.crt 文件似乎也有效，因为来自我本地计算机 的以下命令 运行 给出了 HTTP 302找到（远程 wget 不支持 --ca-certificate 参数）：
wget --ca-certificate=ca-certificates.crt https://git.tech.rz.blub.de
--2021-04-26 11:03:16-- https://git.tech.rz.blub.de/
Loaded CA certificate 'ca-certificates.crt'
Resolving git.tech.rz.blub.de (git.tech.rz.blub.de)... 10.104.85.137, 10.104.84.192, 10.104.86.175
Connecting to git.tech.rz.blub.de (git.tech.rz.blub.de)|10.104.85.137|:443... connected.
HTTP request sent, awaiting response... 302 Found
当我在特使容器上打开 shell 并尝试通过代理发送 http 请求时，我收到 503 服务不可用：
export http_proxy="localhost:10000"
wget git.tech.rz.blub.de
Connecting to localhost:10000 (127.0.0.1:10000)
wget: server returned error: HTTP/1.1 503 Service Unavailable
如特使日志所示，请求已处理并且 DNS 解析成功，但是 ca 验证失败并出现错误 TLS 错误：268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
可以在此处找到日志：https://pastebin.com/pUZZqhSA
完整的特使配置如下所示：
admin:
access_log_path: /dev/stdout
address:
socket_address: { address: 127.0.0.1, port_value: 9901, protocol: TCP }
static_resources:
secrets:
- name: server_cert
tls_certificate:
certificate_chain:
filename: /etc/cert-volume/server.crt
private_key:
filename: /etc/cert-volume/server.key
- name: validation_context
validation_context:
trusted_ca:
filename: /etc/cert-volume/ca-certificates.crt
listeners:
- name: listener_incoming
address:
socket_address: { address: 0.0.0.0, port_value: 8443 }
filter_chains:
filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/config/" }
route: { cluster: local_config }
- match: { prefix: "/basic/" }
route: { cluster: local }
- match: { prefix: "/" }
route: { cluster: local }
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/stdout
log_format:
json_format:
protocol: "%PROTOCOL%"
response_code: "%RESPONSE_CODE%"
response_code_details: "%RESPONSE_CODE_DETAILS%"
connection_termination_details: "%CONNECTION_TERMINATION_DETAILS%"
message: "%LOCAL_REPLY_BODY%"
response_flags: "%RESPONSE_FLAGS%"
local_reply_body: "%LOCAL_REPLY_BODY%"
http_filters:
- name: envoy.filters.http.router
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificate_sds_secret_configs:
- name: server_cert
- name: listener_outgoing
address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: AUTO
stat_prefix: http_to_https
route_config:
name: local_route
virtual_hosts:
- name: app
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: dynamic_forward_proxy_cluster
http_filters:
- name: envoy.filters.http.dynamic_forward_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig
dns_cache_config:
name: dynamic_forward_proxy_cache_config
dns_lookup_family: V4_ONLY
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: local
connect_timeout: 0.25s
type: STATIC
load_assignment:
cluster_name: local
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8080
- name: local_config
connect_timeout: 0.25s
type: STATIC
load_assignment:
cluster_name: local_config
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8081
- name: dynamic_forward_proxy_cluster
connect_timeout: 1s
lb_policy: CLUSTER_PROVIDED
cluster_type:
name: envoy.clusters.dynamic_forward_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig
dns_cache_config:
name: dynamic_forward_proxy_cache_config
dns_lookup_family: V4_ONLY
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
validation_context_sds_secret_config:
name: validation_context
envoy-admin 端点显示，使用了正确的证书：
wget localhost:9901/certs
Connecting to localhost:9901 (127.0.0.1:9901)
saving to 'certs'
certs 100% |*********************************************| 710 0:00:00 ETA
'certs' saved
/ # cat certs
{
"certificates": [
{
"ca_cert": [
{
"path": "/etc/cert-volume/ca-certificates.crt",
"serial_number": "612a55dd000000000002",
"subject_alt_names": [],
"days_until_expiration": "1305",
"valid_from": "2013-11-21T15:40:58Z",
"expiration_time": "2024-11-21T15:50:58Z"
}
],
"cert_chain": []
},
{
"ca_cert": [],
"cert_chain": [
{
"path": "/etc/cert-volume/server.crt",
"serial_number": "74af53b8463a1714",
"subject_alt_names": [
{
"dns": "*.test.comp.blub.de"
}
],
"days_until_expiration": "173",
"valid_from": "2019-10-18T07:10:38Z",
"expiration_time": "2021-10-17T07:10:38Z"
}
]
}
]
}
有没有人知道，这里出了什么问题？
问题已解决。 ca-certificates.crt 缺少必需的 CA。我不知道为什么 wget 没有抛出异常，但是 openssl 抛出异常...