无法让 kubectl 对 dex 和 gangway 进行身份验证
can't get kubectl to authenticate against dex and gangway
我是 运行 centos7 上 kubernetes 1.16.2 的本地实例。我安装了 cert-manager、dex 和 gangway 并且显然可以正常工作。我遇到的问题是 kubectl。
所以我在它自己的入口上有 dex 运行 使用 letsencrypt 证书和 ldap 在它的后端到我们的本地身份验证。和舷梯配置为使用这个 dex 实例。
我收集了一个 letsencrypt.pem CA
curl https://letsencrypt.org/certs/isrgrootx1.pem.txt > isrgrootx1.pem.txt
curl https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt > lets-encrypt-x3-cross-signed.pem.txt
cat isrgrootx1.pem.txt lets-encrypt-x3-cross-signed.pem.txt > /etc/pki/ca-trust/source/anchors/letsencrypt.pem
dex，配置了这些额外的 nginx 入口注释，
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
似乎工作正常：
$ curl -v -L --cacert /etc/pki/ca-trust/source/anchors/letsencrypt.pem https://dex.mydomain.com/.well-known/openid-configuration
* About to connect() to dex.mydomain.com port 443 (#0)
* Trying x.y.z.a...
* Connected to dex.mydomain.com (x.y.z.a) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/ca-trust/source/anchors/letsencrypt.pem
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=dex.mydomain.com
* start date: Oct 27 04:33:31 2019 GMT
* expire date: Jan 25 04:33:31 2020 GMT
* common name: dex.mydomain.com
* issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
> GET /.well-known/openid-configuration HTTP/1.1
> User-Agent: curl/7.29.0
> Host: dex.mydomain.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: openresty/1.15.8.2
< Date: Sun, 27 Oct 2019 06:55:30 GMT
< Content-Type: application/json
< Content-Length: 774
< Connection: keep-alive
< Vary: Accept-Encoding
< Strict-Transport-Security: max-age=15724800; includeSubDomains
<
{
"issuer": "https://dex.mydomain.com",
"authorization_endpoint": "https://dex.mydomain.com/auth",
"token_endpoint": "https://dex.mydomain.com/token",
"jwks_uri": "https://dex.mydomain.com/keys",
"userinfo_endpoint": "https://dex.mydomain.com/userinfo",
"response_types_supported": [
"code"
],
"subject_types_supported": [
"public"
],
"id_token_signing_alg_values_supported": [
"RS256"
],
"scopes_supported": [
"openid",
"email",
"groups",
"profile",
"offline_access"
],
"token_endpoint_auth_methods_supported": [
"client_secret_basic"
],
"claims_supported": [
"aud",
"email",
"email_verified",
"exp",
"iat",
"iss",
"locale",
"name",
"sub"
]
* Connection #0 to host dex.mydomain.com left intact
然后我通过添加以下 containers.command 行在我的 kubernetes 主节点上编辑我的 kubeapi 清单：
- --oidc-issuer-url=https://dex.mydomain.com
- --oidc-ca-file=/etc/pki/ca-trust/source/anchors/letsencrypt.pem
- --oidc-client-id=oidc-auth-client
- --oidc-username-claim=email
- --oidc-groups-claim=groups
我等待 kubernetes master pods 自己重启。一切恢复正常。我再次检查 /etc/pki/ca-trust/source/anchors/letsencrypt.pem 是否可以在 kube-apiserver-k8s pod 中访问。
我配置舷梯 没有 trustedCAPath 然后我继续转到我的舷梯入口，我看到了 dex 身份验证页面。我选择 LDAP，输入我的企业帐户详细信息，然后 gangway 继续进行预期的操作：
echo "-----BEGIN CERTIFICATE-----
MIICyDCCAbCgAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTE5MTAxNDE5MTAyMFoXDTI5MTAxMTE5MTAyMFowFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJac
bCrqxhyd70KzLyvFZ9Ew6kZfrcV7LVI73jdP2Tr9wstE3T3SgwHF8B23zJTFXqjv
W1wGBIIXu/GSIsGuotkyggUMrXDJgakm+SeNeIODsqycJQbyIvnUy++PHGPXHU+p
Wuh8fvNuK51kwbn663ZFZDRrRoL3/zptrhOm3dgAomCqW8BBrQWeW1lOIk2M8NOR
6iqkJ97nSmUAMM0pKGbyQcy3xHFjPr9nLjInC+R70DoaIZtB4sDeBkHpTkVGOEDn
FacnxkhcXt/LYGZNMjJKbrpKIlx8TVSTPR57AfIRfQsnJ59p6M0pnY5TFm2eGd0L
QJG0/GpExZM47PZnwFcCAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAAmc59zdag+ARkBDA55nxGC2qCaY
DDhfR/FUghlMEkiBs+Ys8o0ndnL/2lorcFOiquikP/fV6SMMZOr17OQw+C0Hh6kb
H6f5Dm8e/Y2C6Ku5ZTvGyU+CYDtHS4nejZ2Z1a8ZxKpDfUisgrFqB42+imCeK7uS
of2WgVLoWftjUNOebsT8di0L1j7vpQPOh+doQqh5kKEQwwJ2sK22NJDRa6rPqQ/t
F5UHkQcrJUdOZq/ZXtqEu2uFsbJc85tEoVK3l2xL4AoZbcE3K0vjgGpxtjk5LwdY
LSrEoraw1hxeFIA7hDBxm71y7eT5ZTB967cLOXsuMe/l4keMgpOy8KHO+xY=
-----END CERTIFICATE-----
" \ > ca-k8s-cluster.pem
kubectl config set-cluster k8s-cluster --server=https://k8s-login.mydomain.com/ --certificate-authority=ca-k8s-cluster.pem --embed-certs
kubectl config set-credentials me@mydomain.com \
--auth-provider=oidc \
--auth-provider-arg='idp-issuer-url=https://dex.mydomain.com' \
--auth-provider-arg='client-id=oidc-auth-client' \
--auth-provider-arg='client-secret=wqTkyx6vtpH56FI9K3DE+IaFuwTVToJ5f3CBAuidTtU=' \
--auth-provider-arg='refresh-token=Chl2bWtrYTJsdnlvZWt4dW9raGc3aGxrcnpxEhlvdmlwcWZ2c3dsZGQ0d3dzd3M3a2ZmNHVq' \
--auth-provider-arg='id-token=eyJhbGciOiJSUzI1NiIsImtpZCI6ImFlZDFlMzYxY2NmMjY4NzI3MDVjYTBmNDcxYzFlMDgzZGQ1MjA4YWUifQ.eyJpc3MiOiJodHRwczovL2RleC5zbGFjLnN0YW5mb3JkLmVkdSIsInN1YiI6IkNnTjVkR3dTQkd4a1lYQSIsImF1ZCI6Im9pZGMtYXV0aC1jbGllbnQiLCJleHAiOjE1NzIyNDQ3ODYsImlhdCI6MTU3MjE1ODM4NiwiYXRfaGFzaCI6IlVEdUxfNFdGb3JTRTZhSzNzY0wtQ0EiLCJlbWFpbCI6Inl0bEBzbGFjLnN0YW5mb3JkLmVkdSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoiWWVlIFRpbmcgTGkifQ.Qyjf_IXxyHX7-Z8uLHNxCfC7ne0fZfCk4XoyVD_tom-b9bLf72-p1IxW8HPLo6QgCcnH9QFw3X4XKpFU7sjXZTwu59CxkCfQSG1XKHIKrzUbX7g3c4ZX1b7r7tTnAmGXTj-nG15crYnBcJjgFJRac_qKjqJUBED1_RUjKXzjpQaSlhmsnu8roRVcUJvHnsORHZlSF2kOYlLFlW2fi8cYa3JXISBhY2vzUYJmwopFF8LP3C_GrzrjEsql0984yFDHYNGKtnVhBRCFDBEmBEc9OgxEi8OXuN93pKu-ApNiagaoVBprgWzgl2eijogkCrHn_T7NcXBpx211V1YzXs_-_w'
kubectl config set-context k8s-cluster --cluster=k8s-cluster --user=me@mydomain.com
kubectl config use-context k8s-cluster
rm ca-k8s-cluster.pem
我尽职尽责地粘贴到我的终端。然而，当我尝试（没有为自己设置任何 RBAC 角色）时，我得到
$ kubectl get pods
Unable to connect to the server: x509: certificate signed by unknown authority
我也尝试从 kubeapi 中删除 --oidc-ca-file，但我得到了同样的错误。
我也试过添加
--auth-provider-arg='idp-certificate-authority=/etc/pki/ca-trust/source/anchors/letsencrypt.pem'
我的 kubectl config 凭据 - 但无济于事。
我也试过：
$ kubectl --insecure-skip-tls-verify=true get pods
error: yaml: line 13: mapping values are not allowed in this context
我在 kupeapi 日志中没有看到任何异常，我完全不知道接下来要尝试什么。有什么建议吗？
grrr....我解决了。全部在行中：
kubectl config set-cluster k8s-cluster --server=https://k8s-login.mydomain.com/ --certificate-authority=ca-k8s-cluster.pem --embed-certs
有问题的 --server 应该是 kubeapi 端点，而不是我的舷梯服务器！我将其追溯到舷梯配置图中我的 apiServerURL 的重复。
我是 运行 centos7 上 kubernetes 1.16.2 的本地实例。我安装了 cert-manager、dex 和 gangway 并且显然可以正常工作。我遇到的问题是 kubectl。
所以我在它自己的入口上有 dex 运行 使用 letsencrypt 证书和 ldap 在它的后端到我们的本地身份验证。和舷梯配置为使用这个 dex 实例。
我收集了一个 letsencrypt.pem CA
curl https://letsencrypt.org/certs/isrgrootx1.pem.txt > isrgrootx1.pem.txt
curl https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt > lets-encrypt-x3-cross-signed.pem.txt
cat isrgrootx1.pem.txt lets-encrypt-x3-cross-signed.pem.txt > /etc/pki/ca-trust/source/anchors/letsencrypt.pem
dex，配置了这些额外的 nginx 入口注释，
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
似乎工作正常：
$ curl -v -L --cacert /etc/pki/ca-trust/source/anchors/letsencrypt.pem https://dex.mydomain.com/.well-known/openid-configuration
* About to connect() to dex.mydomain.com port 443 (#0)
* Trying x.y.z.a...
* Connected to dex.mydomain.com (x.y.z.a) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/ca-trust/source/anchors/letsencrypt.pem
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=dex.mydomain.com
* start date: Oct 27 04:33:31 2019 GMT
* expire date: Jan 25 04:33:31 2020 GMT
* common name: dex.mydomain.com
* issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
> GET /.well-known/openid-configuration HTTP/1.1
> User-Agent: curl/7.29.0
> Host: dex.mydomain.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: openresty/1.15.8.2
< Date: Sun, 27 Oct 2019 06:55:30 GMT
< Content-Type: application/json
< Content-Length: 774
< Connection: keep-alive
< Vary: Accept-Encoding
< Strict-Transport-Security: max-age=15724800; includeSubDomains
<
{
"issuer": "https://dex.mydomain.com",
"authorization_endpoint": "https://dex.mydomain.com/auth",
"token_endpoint": "https://dex.mydomain.com/token",
"jwks_uri": "https://dex.mydomain.com/keys",
"userinfo_endpoint": "https://dex.mydomain.com/userinfo",
"response_types_supported": [
"code"
],
"subject_types_supported": [
"public"
],
"id_token_signing_alg_values_supported": [
"RS256"
],
"scopes_supported": [
"openid",
"email",
"groups",
"profile",
"offline_access"
],
"token_endpoint_auth_methods_supported": [
"client_secret_basic"
],
"claims_supported": [
"aud",
"email",
"email_verified",
"exp",
"iat",
"iss",
"locale",
"name",
"sub"
]
* Connection #0 to host dex.mydomain.com left intact
然后我通过添加以下 containers.command 行在我的 kubernetes 主节点上编辑我的 kubeapi 清单：
- --oidc-issuer-url=https://dex.mydomain.com
- --oidc-ca-file=/etc/pki/ca-trust/source/anchors/letsencrypt.pem
- --oidc-client-id=oidc-auth-client
- --oidc-username-claim=email
- --oidc-groups-claim=groups
我等待 kubernetes master pods 自己重启。一切恢复正常。我再次检查 /etc/pki/ca-trust/source/anchors/letsencrypt.pem 是否可以在 kube-apiserver-k8s pod 中访问。
我配置舷梯 没有 trustedCAPath 然后我继续转到我的舷梯入口，我看到了 dex 身份验证页面。我选择 LDAP，输入我的企业帐户详细信息，然后 gangway 继续进行预期的操作：
echo "-----BEGIN CERTIFICATE-----
MIICyDCCAbCgAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTE5MTAxNDE5MTAyMFoXDTI5MTAxMTE5MTAyMFowFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJac
bCrqxhyd70KzLyvFZ9Ew6kZfrcV7LVI73jdP2Tr9wstE3T3SgwHF8B23zJTFXqjv
W1wGBIIXu/GSIsGuotkyggUMrXDJgakm+SeNeIODsqycJQbyIvnUy++PHGPXHU+p
Wuh8fvNuK51kwbn663ZFZDRrRoL3/zptrhOm3dgAomCqW8BBrQWeW1lOIk2M8NOR
6iqkJ97nSmUAMM0pKGbyQcy3xHFjPr9nLjInC+R70DoaIZtB4sDeBkHpTkVGOEDn
FacnxkhcXt/LYGZNMjJKbrpKIlx8TVSTPR57AfIRfQsnJ59p6M0pnY5TFm2eGd0L
QJG0/GpExZM47PZnwFcCAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAAmc59zdag+ARkBDA55nxGC2qCaY
DDhfR/FUghlMEkiBs+Ys8o0ndnL/2lorcFOiquikP/fV6SMMZOr17OQw+C0Hh6kb
H6f5Dm8e/Y2C6Ku5ZTvGyU+CYDtHS4nejZ2Z1a8ZxKpDfUisgrFqB42+imCeK7uS
of2WgVLoWftjUNOebsT8di0L1j7vpQPOh+doQqh5kKEQwwJ2sK22NJDRa6rPqQ/t
F5UHkQcrJUdOZq/ZXtqEu2uFsbJc85tEoVK3l2xL4AoZbcE3K0vjgGpxtjk5LwdY
LSrEoraw1hxeFIA7hDBxm71y7eT5ZTB967cLOXsuMe/l4keMgpOy8KHO+xY=
-----END CERTIFICATE-----
" \ > ca-k8s-cluster.pem
kubectl config set-cluster k8s-cluster --server=https://k8s-login.mydomain.com/ --certificate-authority=ca-k8s-cluster.pem --embed-certs
kubectl config set-credentials me@mydomain.com \
--auth-provider=oidc \
--auth-provider-arg='idp-issuer-url=https://dex.mydomain.com' \
--auth-provider-arg='client-id=oidc-auth-client' \
--auth-provider-arg='client-secret=wqTkyx6vtpH56FI9K3DE+IaFuwTVToJ5f3CBAuidTtU=' \
--auth-provider-arg='refresh-token=Chl2bWtrYTJsdnlvZWt4dW9raGc3aGxrcnpxEhlvdmlwcWZ2c3dsZGQ0d3dzd3M3a2ZmNHVq' \
--auth-provider-arg='id-token=eyJhbGciOiJSUzI1NiIsImtpZCI6ImFlZDFlMzYxY2NmMjY4NzI3MDVjYTBmNDcxYzFlMDgzZGQ1MjA4YWUifQ.eyJpc3MiOiJodHRwczovL2RleC5zbGFjLnN0YW5mb3JkLmVkdSIsInN1YiI6IkNnTjVkR3dTQkd4a1lYQSIsImF1ZCI6Im9pZGMtYXV0aC1jbGllbnQiLCJleHAiOjE1NzIyNDQ3ODYsImlhdCI6MTU3MjE1ODM4NiwiYXRfaGFzaCI6IlVEdUxfNFdGb3JTRTZhSzNzY0wtQ0EiLCJlbWFpbCI6Inl0bEBzbGFjLnN0YW5mb3JkLmVkdSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoiWWVlIFRpbmcgTGkifQ.Qyjf_IXxyHX7-Z8uLHNxCfC7ne0fZfCk4XoyVD_tom-b9bLf72-p1IxW8HPLo6QgCcnH9QFw3X4XKpFU7sjXZTwu59CxkCfQSG1XKHIKrzUbX7g3c4ZX1b7r7tTnAmGXTj-nG15crYnBcJjgFJRac_qKjqJUBED1_RUjKXzjpQaSlhmsnu8roRVcUJvHnsORHZlSF2kOYlLFlW2fi8cYa3JXISBhY2vzUYJmwopFF8LP3C_GrzrjEsql0984yFDHYNGKtnVhBRCFDBEmBEc9OgxEi8OXuN93pKu-ApNiagaoVBprgWzgl2eijogkCrHn_T7NcXBpx211V1YzXs_-_w'
kubectl config set-context k8s-cluster --cluster=k8s-cluster --user=me@mydomain.com
kubectl config use-context k8s-cluster
rm ca-k8s-cluster.pem
我尽职尽责地粘贴到我的终端。然而，当我尝试（没有为自己设置任何 RBAC 角色）时，我得到
$ kubectl get pods
Unable to connect to the server: x509: certificate signed by unknown authority
我也尝试从 kubeapi 中删除 --oidc-ca-file，但我得到了同样的错误。
我也试过添加
--auth-provider-arg='idp-certificate-authority=/etc/pki/ca-trust/source/anchors/letsencrypt.pem'
我的 kubectl config 凭据 - 但无济于事。
我也试过：
$ kubectl --insecure-skip-tls-verify=true get pods
error: yaml: line 13: mapping values are not allowed in this context
我在 kupeapi 日志中没有看到任何异常，我完全不知道接下来要尝试什么。有什么建议吗？
grrr....我解决了。全部在行中：
kubectl config set-cluster k8s-cluster --server=https://k8s-login.mydomain.com/ --certificate-authority=ca-k8s-cluster.pem --embed-certs
有问题的 --server 应该是 kubeapi 端点，而不是我的舷梯服务器！我将其追溯到舷梯配置图中我的 apiServerURL 的重复。