Kubernetes RBAC - 禁止尝试授予额外权限
Kubernetes RBAC - forbidden attempt to grant extra privileges
我在自定义 CoreOS
集群上使用 Kubernetes
v1.8.14:
$ kubectl version --short
Client Version: v1.10.5
Server Version: v1.8.14+coreos.0
尝试创建以下内容时 ClusterRole
:
$ cat ClusterRole.yml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:coredns
rules:
- apiGroups:
- ""
resources:
- endpoints
- services
- pods
- namespaces
verbs:
- list
- watch
我收到以下错误:
$ kubectl create -f ClusterRole.yml
Error from server (Forbidden): error when creating "ClusterRole.yml": clusterroles.rbac.authorization.k8s.io "system:coredns" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["watch"]}] user=&{cluster-admin [system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]
据我所知,我正在连接 cluster-admin
,因此应该有足够的权限来实现我想要实现的目标。以下是相关的 cluster-admin
配置:
$ cat ~/.kube/config
apiVersion: v1
kind: Config
current-context: dev
preferences:
colors: true
clusters:
- cluster:
certificate-authority: cluster-ca.pem
server: https://k8s.loc:4430
name: dev
contexts:
- context:
cluster: dev
namespace: kube-system
user: cluster-admin
name: dev
users:
- name: cluster-admin
user:
client-certificate: cluster.pem
client-key: cluster-key.pem
$ kubectl get clusterrole cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: 2018-07-30T14:44:44Z
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: cluster-admin
resourceVersion: "1164791"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/cluster-admin
uid: 196ffecc-9407-11e8-bd67-525400ac0b7d
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'
$ kubectl get clusterrolebinding cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: 2018-07-30T14:44:45Z
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: cluster-admin
resourceVersion: "1164832"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cluster-admin
uid: 19e516a6-9407-11e8-bd67-525400ac0b7d
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:masters
$ kubectl get serviceaccount cluster-admin -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: 2018-07-30T13:32:13Z
name: cluster-admin
namespace: kube-system
resourceVersion: "1158783"
selfLink: /api/v1/namespaces/kube-system/serviceaccounts/cluster-admin
uid: f809e079-93fc-11e8-8b85-525400546bcd
secrets:
- name: cluster-admin-token-t7s4c
我知道这是 RBAC 问题,但不知道如何进一步调试它。
编辑-1.
我尝试了建议,不幸的是没有快乐...
$ kubectl get clusterrolebinding cluster-admin-binding -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: 2018-07-31T09:21:34Z
name: cluster-admin-binding
resourceVersion: "1252260"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cluster-admin-binding
uid: 1e1c0647-94a3-11e8-9f9b-525400ac0b7d
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: cluster-admin
namespace: default
$ kubectl describe secret $(kubectl get secret | awk '/cluster-admin/{print }')
Name: cluster-admin-token-t7s4c
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name=cluster-admin
kubernetes.io/service-account.uid=f809e079-93fc-11e8-8b85-525400546bcd
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1785 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVyLWFkbWluLXRva2VuLXQ3czRjIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXItYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJmODA5ZTA3OS05M2ZjLTExZTgtOGI4NS01MjU0MDA1NDZiY2QiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06Y2x1c3Rlci1hZG1pbiJ9.rC1x9Or8GArkhC3P0s-l_Pc0e6TEUwfbJtXAN2w-cOaRUCNCo6r4WxXKu32ngOg86TXqCho2wBopXtbJ2CparIb7FWDXzri6O6LPFzHWNzZo3b-TON2yxHMWECGjpbbqjDgkPKDEldkdxJehDBJM_GFAaUdNyYpFFsP1_t3vVIsf2DpCjeMlOBSprYRcEKmDiE6ehF4RSn1JqB7TVpvTZ_WAL4CRZoTJtZDVoF75AtKIADtVXTxVv_ewznDCKUWDupg5Jk44QSMJ0YiG30QYYM699L5iFLirzD5pj0EEPAoMeOqSjdp7KvDzIM2tBiu8YYl6Fj7pG_53WjZrvlSk5pgPLS-jPKOkixFM9FfB2eeuP0eWwLO5wvU5s--a2ekkEhaqHTXgigeedudDA_5JVIJTS0m6V9gcbE4_kYRpU7_QD_0TR68C5yxUL83KfOzj6A_S6idOZ-p7Ni6ffE_KlGqqcgUUR2MTakJgimjn0gYHNaIqmHIu4YhrT-jffP0-5ZClbI5srj-aB4YqGtCH9w5_KBYD4S2y6Rjv4kO00nZyvi0jAHlZ6el63TQPWYkjyPL2moF_P8xcPeoDrF6o8bXDzFqlXLqda2Nqyo8LMhLxjpe_wFeGuwzIUxwwtH1RUR6BISRUf86041aa2PeJMqjTfaU0u_SvO-yHMGxZt3o
然后修改~/.kube/config
:
$ cat ~/.kube/config
apiVersion: v1
kind: Config
current-context: dev
preferences:
colors: true
clusters:
- cluster:
certificate-authority: cluster-ca.pem
server: https://k8s.loc:4430
name: dev
contexts:
- context:
cluster: dev
namespace: kube-system
user: cluster-admin-2
name: dev
users:
- name: cluster-admin
user:
client-certificate: cluster.pem
client-key: cluster-key.pem
- name: cluster-admin-2
user:
token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVyLWFkbWluLXRva2VuLXQ3czRjIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXItYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJmODA5ZTA3OS05M2ZjLTExZTgtOGI4NS01MjU0MDA1NDZiY2QiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06Y2x1c3Rlci1hZG1pbiJ9.rC1x9Or8GArkhC3P0s-l_Pc0e6TEUwfbJtXAN2w-cOaRUCNCo6r4WxXKu32ngOg86TXqCho2wBopXtbJ2CparIb7FWDXzri6O6LPFzHWNzZo3b-TON2yxHMWECGjpbbqjDgkPKDEldkdxJehDBJM_GFAaUdNyYpFFsP1_t3vVIsf2DpCjeMlOBSprYRcEKmDiE6ehF4RSn1JqB7TVpvTZ_WAL4CRZoTJtZDVoF75AtKIADtVXTxVv_ewznDCKUWDupg5Jk44QSMJ0YiG30QYYM699L5iFLirzD5pj0EEPAoMeOqSjdp7KvDzIM2tBiu8YYl6Fj7pG_53WjZrvlSk5pgPLS-jPKOkixFM9FfB2eeuP0eWwLO5wvU5s--a2ekkEhaqHTXgigeedudDA_5JVIJTS0m6V9gcbE4_kYRpU7_QD_0TR68C5yxUL83KfOzj6A_S6idOZ-p7Ni6ffE_KlGqqcgUUR2MTakJgimjn0gYHNaIqmHIu4YhrT-jffP0-5ZClbI5srj-aB4YqGtCH9w5_KBYD4S2y6Rjv4kO00nZyvi0jAHlZ6el63TQPWYkjyPL2moF_P8xcPeoDrF6o8bXDzFqlXLqda2Nqyo8LMhLxjpe_wFeGuwzIUxwwtH1RUR6BISRUf86041aa2PeJMqjTfaU0u_SvO-yHMGxZt3o
然后尝试应用相同的 ClusterRole
,结果出现相同的错误:
$ kubectl apply -f ClusterRole.yml
Error from server (Forbidden): error when creating "ClusterRole.yml": clusterroles.rbac.authorization.k8s.io "system:coredns" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["watch"]}] user=&{system:serviceaccount:kube-system:cluster-admin f809e079-93fc-11e8-8b85-525400546bcd [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]
下面是我用来启动的标志 apiserver
:
containers:
- name: kube-apiserver
image: quay.io/coreos/hyperkube:${K8S_VER}
command:
- /hyperkube
- apiserver
- --bind-address=0.0.0.0
- --etcd-servers=${ETCD_ENDPOINTS}
- --allow-privileged=true
- --service-cluster-ip-range=${SERVICE_IP_RANGE}
- --secure-port=443
- --advertise-address=${ADVERTISE_IP}
- --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota
- --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem
- --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
- --client-ca-file=/etc/kubernetes/ssl/ca.pem
- --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem
- --runtime-config=extensions/v1beta1/networkpolicies=true
- --anonymous-auth=false
- --authorization-mode=AlwaysAllow,RBAC,Node
下面是我用来生成 tls
证书的脚本:
root ca:
openssl genrsa -out ca-key.pem 4096
openssl req -x509 -new -nodes -key ca-key.pem -days 3650 -out ca.pem -subj "/CN=kube-ca"
apiserver:
cat > openssl.cnf <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
DNS.5 = ${MASTER_LB_DNS}
IP.1 = ${K8S_SERVICE_IP}
IP.2 = ${MASTER_HOST}
EOF
openssl genrsa -out apiserver-key.pem 4096
openssl req -new -key apiserver-key.pem -out apiserver.csr -subj "/CN=kube-apiserver" -config openssl.cnf
openssl x509 -req -in apiserver.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out apiserver.pem -days 3650 -extensions v3_req -extfile openssl.cnf
集群管理员:
openssl genrsa -out cluster-admin-key.pem 4096
openssl req -new -key cluster-admin-key.pem -out cluster-admin.csr -subj "/CN=cluster-admin"
openssl x509 -req -in cluster-admin.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out cluster-admin.pem -days 3650
我希望这能让您更深入地了解我的系统出了什么问题。
编辑-2.
我注意到我的系统配置与@MarcinRomaszewicz 建议的略有差异,因此 cluster-admin 的 namespace ServiceAccount
,在我的例子中它在 kube-system
而不是 default
命名空间 :
$ kubectl delete clusterrolebinding cluster-admin-binding
clusterrolebinding.rbac.authorization.k8s.io "cluster-admin-binding" deleted
$ kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole=cluster-admin --serviceaccount=kube-system:cluster-admin
clusterrolebinding.rbac.authorization.k8s.io "cluster-admin-binding" created
$ kubectl apply -f ClusterRole.yml
clusterrole.rbac.authorization.k8s.io "system:coredns" created
但是它仍然不适用于我的证书...
编辑-3.
如评论中所建议,为了让 apiserver
将用户识别为 cluster-admin
,该用户证书中的主题行必须包含以下项目:Subject: CN = cluster-admin, O = system:masters
.生成此类证书的一种方法如下:
openssl genrsa -out cluster-admin-key.pem 4096
openssl req -new -key cluster-admin-key.pem -out cluster-admin.csr -subj "/CN=cluster-admin/O=system:masters"
openssl x509 -req -in cluster-admin.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out cluster-admin.pem -days 3650
这里没有足够的信息来回答您的问题。
听起来您正在 运行 防止权限升级:
https://kubernetes.io/docs/reference/access-authn-authz/rbac/#privilege-escalation-prevention-and-bootstrapping
这意味着您实际上 运行 不是集群管理员。检查你的 kubectl 配置。例如,您可能 运行 因为 "admin" 受限于特定名称 space。
(根据下面的评论进行编辑)
您在 k8s 中的身份是由 cluster.pem 证书的内容确定的,而不是 kubeconfig 中的用户名,因为该用户名仅在 kubeconfig 文件中有效。您的实际用户由该证书确定。
我看到您有一个名为 cluster-admin 的服务帐户,但它不是 "system:masters" 的成员,因为组是对用户进行身份验证的身份验证系统的 属性 - 您需要创建一个明确的集群角色绑定,以将您的集群管理员服务帐户绑定到集群管理员集群角色。
kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --serviceaccount=default:cluster-admin
您应该会看到 clusterrole 现在与您的服务帐户绑定了。
$ kubectl get clusterrolebinding cluster-admin-binding -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: 2018-07-30T22:02:33Z
name: cluster-admin-binding
resourceVersion: "71152"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cluster-admin-binding
uid: 42a2862c-9444-11e8-8b71-080027de17da
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: cluster-admin
namespace: default
请注意底部,绑定适用于 "ServiceAccount",而不适用于群组。
您的服务帐户有一个访问令牌,使用它而不是您的证书进行身份验证。我为自己创建了一个集群管理服务帐户,这就是我获取令牌的方式:
$ kubectl describe secret $(kubectl get secret | grep cluster-admin | awk '{print }')
Name: cluster-admin-token-96vdz
Namespace: default
Labels: <none>
Annotations: kubernetes.io/service-account.name=cluster-admin
kubernetes.io/service-account.uid=f872f08b-9442-11e8-8b71-080027de17da
Type: kubernetes.io/service-account-token
Data
====
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImNsdXN0ZXItYWRtaW4tdG9rZW4tOTZ2ZHoiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiY2x1c3Rlci1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImY4NzJmMDhiLTk0NDItMTFlOC04YjcxLTA4MDAyN2RlMTdkYSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmNsdXN0ZXItYWRtaW4ifQ.<signature snipped>
ca.crt: 1066 bytes
namespace: 7 bytes
更新 kubeconfig 以使用该令牌而不是您当前使用的证书对您自己进行身份验证,并且您应该成功通过该集群管理服务帐户的身份验证。
(编辑 2)
事实证明,用于对 Kubernetes 进行身份验证的证书没有关于用户的任何身份声明。 Kubernetes 依靠身份验证模块对用户进行身份验证,在这种情况下,基于证书。它期望证书包含将用户放入 "system:masters" 组的声明,方法是将组织设置为 "system:masters"。
这里有很多动人的片段。该问题与服务帐户或角色无关,而是与用户身份验证有关,这是非常不透明的。
我在自定义 CoreOS
集群上使用 Kubernetes
v1.8.14:
$ kubectl version --short
Client Version: v1.10.5
Server Version: v1.8.14+coreos.0
尝试创建以下内容时 ClusterRole
:
$ cat ClusterRole.yml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:coredns
rules:
- apiGroups:
- ""
resources:
- endpoints
- services
- pods
- namespaces
verbs:
- list
- watch
我收到以下错误:
$ kubectl create -f ClusterRole.yml
Error from server (Forbidden): error when creating "ClusterRole.yml": clusterroles.rbac.authorization.k8s.io "system:coredns" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["watch"]}] user=&{cluster-admin [system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]
据我所知,我正在连接 cluster-admin
,因此应该有足够的权限来实现我想要实现的目标。以下是相关的 cluster-admin
配置:
$ cat ~/.kube/config
apiVersion: v1
kind: Config
current-context: dev
preferences:
colors: true
clusters:
- cluster:
certificate-authority: cluster-ca.pem
server: https://k8s.loc:4430
name: dev
contexts:
- context:
cluster: dev
namespace: kube-system
user: cluster-admin
name: dev
users:
- name: cluster-admin
user:
client-certificate: cluster.pem
client-key: cluster-key.pem
$ kubectl get clusterrole cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: 2018-07-30T14:44:44Z
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: cluster-admin
resourceVersion: "1164791"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/cluster-admin
uid: 196ffecc-9407-11e8-bd67-525400ac0b7d
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'
$ kubectl get clusterrolebinding cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: 2018-07-30T14:44:45Z
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: cluster-admin
resourceVersion: "1164832"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cluster-admin
uid: 19e516a6-9407-11e8-bd67-525400ac0b7d
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:masters
$ kubectl get serviceaccount cluster-admin -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: 2018-07-30T13:32:13Z
name: cluster-admin
namespace: kube-system
resourceVersion: "1158783"
selfLink: /api/v1/namespaces/kube-system/serviceaccounts/cluster-admin
uid: f809e079-93fc-11e8-8b85-525400546bcd
secrets:
- name: cluster-admin-token-t7s4c
我知道这是 RBAC 问题,但不知道如何进一步调试它。
编辑-1.
我尝试了建议,不幸的是没有快乐...
$ kubectl get clusterrolebinding cluster-admin-binding -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: 2018-07-31T09:21:34Z
name: cluster-admin-binding
resourceVersion: "1252260"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cluster-admin-binding
uid: 1e1c0647-94a3-11e8-9f9b-525400ac0b7d
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: cluster-admin
namespace: default
$ kubectl describe secret $(kubectl get secret | awk '/cluster-admin/{print }')
Name: cluster-admin-token-t7s4c
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name=cluster-admin
kubernetes.io/service-account.uid=f809e079-93fc-11e8-8b85-525400546bcd
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1785 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVyLWFkbWluLXRva2VuLXQ3czRjIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXItYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJmODA5ZTA3OS05M2ZjLTExZTgtOGI4NS01MjU0MDA1NDZiY2QiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06Y2x1c3Rlci1hZG1pbiJ9.rC1x9Or8GArkhC3P0s-l_Pc0e6TEUwfbJtXAN2w-cOaRUCNCo6r4WxXKu32ngOg86TXqCho2wBopXtbJ2CparIb7FWDXzri6O6LPFzHWNzZo3b-TON2yxHMWECGjpbbqjDgkPKDEldkdxJehDBJM_GFAaUdNyYpFFsP1_t3vVIsf2DpCjeMlOBSprYRcEKmDiE6ehF4RSn1JqB7TVpvTZ_WAL4CRZoTJtZDVoF75AtKIADtVXTxVv_ewznDCKUWDupg5Jk44QSMJ0YiG30QYYM699L5iFLirzD5pj0EEPAoMeOqSjdp7KvDzIM2tBiu8YYl6Fj7pG_53WjZrvlSk5pgPLS-jPKOkixFM9FfB2eeuP0eWwLO5wvU5s--a2ekkEhaqHTXgigeedudDA_5JVIJTS0m6V9gcbE4_kYRpU7_QD_0TR68C5yxUL83KfOzj6A_S6idOZ-p7Ni6ffE_KlGqqcgUUR2MTakJgimjn0gYHNaIqmHIu4YhrT-jffP0-5ZClbI5srj-aB4YqGtCH9w5_KBYD4S2y6Rjv4kO00nZyvi0jAHlZ6el63TQPWYkjyPL2moF_P8xcPeoDrF6o8bXDzFqlXLqda2Nqyo8LMhLxjpe_wFeGuwzIUxwwtH1RUR6BISRUf86041aa2PeJMqjTfaU0u_SvO-yHMGxZt3o
然后修改~/.kube/config
:
$ cat ~/.kube/config
apiVersion: v1
kind: Config
current-context: dev
preferences:
colors: true
clusters:
- cluster:
certificate-authority: cluster-ca.pem
server: https://k8s.loc:4430
name: dev
contexts:
- context:
cluster: dev
namespace: kube-system
user: cluster-admin-2
name: dev
users:
- name: cluster-admin
user:
client-certificate: cluster.pem
client-key: cluster-key.pem
- name: cluster-admin-2
user:
token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVyLWFkbWluLXRva2VuLXQ3czRjIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXItYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJmODA5ZTA3OS05M2ZjLTExZTgtOGI4NS01MjU0MDA1NDZiY2QiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06Y2x1c3Rlci1hZG1pbiJ9.rC1x9Or8GArkhC3P0s-l_Pc0e6TEUwfbJtXAN2w-cOaRUCNCo6r4WxXKu32ngOg86TXqCho2wBopXtbJ2CparIb7FWDXzri6O6LPFzHWNzZo3b-TON2yxHMWECGjpbbqjDgkPKDEldkdxJehDBJM_GFAaUdNyYpFFsP1_t3vVIsf2DpCjeMlOBSprYRcEKmDiE6ehF4RSn1JqB7TVpvTZ_WAL4CRZoTJtZDVoF75AtKIADtVXTxVv_ewznDCKUWDupg5Jk44QSMJ0YiG30QYYM699L5iFLirzD5pj0EEPAoMeOqSjdp7KvDzIM2tBiu8YYl6Fj7pG_53WjZrvlSk5pgPLS-jPKOkixFM9FfB2eeuP0eWwLO5wvU5s--a2ekkEhaqHTXgigeedudDA_5JVIJTS0m6V9gcbE4_kYRpU7_QD_0TR68C5yxUL83KfOzj6A_S6idOZ-p7Ni6ffE_KlGqqcgUUR2MTakJgimjn0gYHNaIqmHIu4YhrT-jffP0-5ZClbI5srj-aB4YqGtCH9w5_KBYD4S2y6Rjv4kO00nZyvi0jAHlZ6el63TQPWYkjyPL2moF_P8xcPeoDrF6o8bXDzFqlXLqda2Nqyo8LMhLxjpe_wFeGuwzIUxwwtH1RUR6BISRUf86041aa2PeJMqjTfaU0u_SvO-yHMGxZt3o
然后尝试应用相同的 ClusterRole
,结果出现相同的错误:
$ kubectl apply -f ClusterRole.yml
Error from server (Forbidden): error when creating "ClusterRole.yml": clusterroles.rbac.authorization.k8s.io "system:coredns" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["watch"]}] user=&{system:serviceaccount:kube-system:cluster-admin f809e079-93fc-11e8-8b85-525400546bcd [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]
下面是我用来启动的标志 apiserver
:
containers:
- name: kube-apiserver
image: quay.io/coreos/hyperkube:${K8S_VER}
command:
- /hyperkube
- apiserver
- --bind-address=0.0.0.0
- --etcd-servers=${ETCD_ENDPOINTS}
- --allow-privileged=true
- --service-cluster-ip-range=${SERVICE_IP_RANGE}
- --secure-port=443
- --advertise-address=${ADVERTISE_IP}
- --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota
- --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem
- --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
- --client-ca-file=/etc/kubernetes/ssl/ca.pem
- --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem
- --runtime-config=extensions/v1beta1/networkpolicies=true
- --anonymous-auth=false
- --authorization-mode=AlwaysAllow,RBAC,Node
下面是我用来生成 tls
证书的脚本:
root ca:
openssl genrsa -out ca-key.pem 4096
openssl req -x509 -new -nodes -key ca-key.pem -days 3650 -out ca.pem -subj "/CN=kube-ca"
apiserver:
cat > openssl.cnf <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
DNS.5 = ${MASTER_LB_DNS}
IP.1 = ${K8S_SERVICE_IP}
IP.2 = ${MASTER_HOST}
EOF
openssl genrsa -out apiserver-key.pem 4096
openssl req -new -key apiserver-key.pem -out apiserver.csr -subj "/CN=kube-apiserver" -config openssl.cnf
openssl x509 -req -in apiserver.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out apiserver.pem -days 3650 -extensions v3_req -extfile openssl.cnf
集群管理员:
openssl genrsa -out cluster-admin-key.pem 4096
openssl req -new -key cluster-admin-key.pem -out cluster-admin.csr -subj "/CN=cluster-admin"
openssl x509 -req -in cluster-admin.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out cluster-admin.pem -days 3650
我希望这能让您更深入地了解我的系统出了什么问题。
编辑-2.
我注意到我的系统配置与@MarcinRomaszewicz 建议的略有差异,因此 cluster-admin 的 namespace ServiceAccount
,在我的例子中它在 kube-system
而不是 default
命名空间 :
$ kubectl delete clusterrolebinding cluster-admin-binding
clusterrolebinding.rbac.authorization.k8s.io "cluster-admin-binding" deleted
$ kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole=cluster-admin --serviceaccount=kube-system:cluster-admin
clusterrolebinding.rbac.authorization.k8s.io "cluster-admin-binding" created
$ kubectl apply -f ClusterRole.yml
clusterrole.rbac.authorization.k8s.io "system:coredns" created
但是它仍然不适用于我的证书...
编辑-3.
如评论中所建议,为了让 apiserver
将用户识别为 cluster-admin
,该用户证书中的主题行必须包含以下项目:Subject: CN = cluster-admin, O = system:masters
.生成此类证书的一种方法如下:
openssl genrsa -out cluster-admin-key.pem 4096
openssl req -new -key cluster-admin-key.pem -out cluster-admin.csr -subj "/CN=cluster-admin/O=system:masters"
openssl x509 -req -in cluster-admin.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out cluster-admin.pem -days 3650
这里没有足够的信息来回答您的问题。
听起来您正在 运行 防止权限升级: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#privilege-escalation-prevention-and-bootstrapping
这意味着您实际上 运行 不是集群管理员。检查你的 kubectl 配置。例如,您可能 运行 因为 "admin" 受限于特定名称 space。
(根据下面的评论进行编辑)
您在 k8s 中的身份是由 cluster.pem 证书的内容确定的,而不是 kubeconfig 中的用户名,因为该用户名仅在 kubeconfig 文件中有效。您的实际用户由该证书确定。
我看到您有一个名为 cluster-admin 的服务帐户,但它不是 "system:masters" 的成员,因为组是对用户进行身份验证的身份验证系统的 属性 - 您需要创建一个明确的集群角色绑定,以将您的集群管理员服务帐户绑定到集群管理员集群角色。
kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --serviceaccount=default:cluster-admin
您应该会看到 clusterrole 现在与您的服务帐户绑定了。
$ kubectl get clusterrolebinding cluster-admin-binding -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: 2018-07-30T22:02:33Z
name: cluster-admin-binding
resourceVersion: "71152"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cluster-admin-binding
uid: 42a2862c-9444-11e8-8b71-080027de17da
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: cluster-admin
namespace: default
请注意底部,绑定适用于 "ServiceAccount",而不适用于群组。
您的服务帐户有一个访问令牌,使用它而不是您的证书进行身份验证。我为自己创建了一个集群管理服务帐户,这就是我获取令牌的方式:
$ kubectl describe secret $(kubectl get secret | grep cluster-admin | awk '{print }')
Name: cluster-admin-token-96vdz
Namespace: default
Labels: <none>
Annotations: kubernetes.io/service-account.name=cluster-admin
kubernetes.io/service-account.uid=f872f08b-9442-11e8-8b71-080027de17da
Type: kubernetes.io/service-account-token
Data
====
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImNsdXN0ZXItYWRtaW4tdG9rZW4tOTZ2ZHoiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiY2x1c3Rlci1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImY4NzJmMDhiLTk0NDItMTFlOC04YjcxLTA4MDAyN2RlMTdkYSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmNsdXN0ZXItYWRtaW4ifQ.<signature snipped>
ca.crt: 1066 bytes
namespace: 7 bytes
更新 kubeconfig 以使用该令牌而不是您当前使用的证书对您自己进行身份验证,并且您应该成功通过该集群管理服务帐户的身份验证。
(编辑 2) 事实证明,用于对 Kubernetes 进行身份验证的证书没有关于用户的任何身份声明。 Kubernetes 依靠身份验证模块对用户进行身份验证,在这种情况下,基于证书。它期望证书包含将用户放入 "system:masters" 组的声明,方法是将组织设置为 "system:masters"。
这里有很多动人的片段。该问题与服务帐户或角色无关,而是与用户身份验证有关,这是非常不透明的。