两个 Kubernetes 集群对 RBAC 的行为不同

Question

我创建了一个应用程序，它需要有权列出、创建、更新和删除不同的 Kubernetes 资源，并且我为它创建了一个 clusterrole，如下所示。我在 Microk8s 上运行的本地 K8s 集群上一切正常，但是当我将它部署在具有相同版本 K8s 的裸机集群上时，我收到错误消息，提示我没有正确的访问权限。

这怎么可能（两者的行为应该相同），有没有办法提前发现这些错误？

我的集群角色：

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: {{ .Release.Namespace }}-cluster-manager-role
rules:
- apiGroups: ["","apps","core", "autoscaling"] # --> I was getting error that I cannot create HPA but after I added "autoscaling" to the apigroup now I can create HPA
  resources: ["*", "namespaces"]
  verbs: ["get", "watch", "list", "patch", "create", "delete", "update"]

# ================
# Current clusterrole on microk8s (which allows me to do all the things)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: "2021-05-31T12:05:58Z"
  name: default-cluster-manager-role
  resourceVersion: "937643"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/default-cluster-manager-role
  uid: 16fb63d6-1261-48a9-bc7f-5c8fffb72c9d
rules:
- apiGroups:
  - ""
  - apps
  - core
  resources:
  - '*'
  - namespaces
  verbs:
  - get
  - watch
  - list
  - patch
  - create
  - delete
  - update

Kubernetes 版本：

# Microk8s
$ kubectl version 
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.15", GitCommit:"2adc8d7091e89b6e3ca8d048140618ec89b39369", GitTreeState:"clean", BuildDate:"2020-09-02T11:31:21Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

# Bare-metal
$ kubectl version 
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.15", GitCommit:"2adc8d7091e89b6e3ca8d048140618ec89b39369", GitTreeState:"clean", BuildDate:"2020-09-02T11:31:21Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

我得到的一些错误：

time="2021-06-22T08:45:31Z" level=error msg="Getting list of PVCs for namespace wws-test failed." func=src/k8s.CreateClusterRole file="/src/k8s/k8s.go:1304"
time="2021-06-22T08:45:31Z" level=error msg="clusterroles.rbac.authorization.k8s.io is forbidden: User \"system:serviceaccount:wws:wws-cluster-manager-sa\" cannot create resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope" func=src/k8s.CreateClusterRole file="/src/k8s/k8s.go:1305"
time="2021-06-22T08:45:31Z" level=error msg="Getting list of PVCs for namespace wws-test failed." func=src/k8s.CreateClusterRoleBinding file="/src/k8s/k8s.go:1232"
time="2021-06-22T08:45:31Z" level=error msg="clusterrolebindings.rbac.authorization.k8s.io is forbidden: User \"system:serviceaccount:wws:wws-cluster-manager-sa\" cannot create resource \"clusterrolebindings\" in API group \"rbac.authorization.k8s.io\" at the cluster scope" func=src/k8s.CreateClusterRoleBinding file="/src/k8s/k8s.go:1233"
time="2021-06-22T08:45:32Z" level=error msg="Getting list of PVCs for namespace wws-test failed." func=src/k8s.CreateRole file="/src/k8s/k8s.go:1448"
time="2021-06-22T08:45:32Z" level=error msg="roles.rbac.authorization.k8s.io is forbidden: User \"system:serviceaccount:wws:wws-cluster-manager-sa\" cannot create resource \"roles\" in API group \"rbac.authorization.k8s.io\" in the namespace \"wws-test\"" func=src/k8s.CreateRole file="/src/k8s/k8s.go:1449"

Answer 1

您应该查看应用于 ServiceAccount 的 ClusterRoleBindings（k get ClusterRoleBinding -o wide）：system:serviceaccount:wws:wws-cluster-manager-sa)

我想在 Minikube 上你的用户可以在你的本地集群上做任何事情。但是，真正的集群不允许您使用默认用户创建新的 ClusterRoles/CluterRoleBindings。

Answer 2

我不知道为什么会这样，但我通过对 apiGroups、resources 和 verbs 的所有三个字段使用 * 解决了这个问题：

rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]

我知道这不是一个干净和完美的解决方案，特别是如果你想更多地控制角色和资源或角色应该访问的动词，但因为没有人（即使我在 Kubernetes 上发布了这个repo github 作为一个问题）知道为什么会这样，我没有时间深入研究这个我接受我自己的回答。

两个 Kubernetes 集群对 RBAC 的行为不同

Two Kubernetes clusters act differently for RBAC

rbac

kubernetes