设置应用程序网关入口控制器（agic）azure kubernetes 服务（aks）的问题
issue setting up app gateway ingress controller(agic) azure kubernetes service(aks)
我已按照以下指南中的步骤在 azure 中设置魔法：
https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/docs/setup/install-existing.md
我有一个 vnet，一个子网中有一个 aks 集群（启用了 rbac），另一个子网中有一个应用程序网关。我已按照使用服务主体和 aad pod 身份授权 ARM 的步骤进行操作。
然而，在这两种情况下，一旦使用 helm-config.yaml 文件安装了入口控制器，pod 的日志就会显示它 运行 但还没有准备好。
以下是使用aad pod身份认证时
kubectl describe pod显示的事件有：
events
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 20m default-scheduler Successfully assigned default/ingress-azure-57bcc69687-bqbdn to aks-agentpool-29530272-vmss000002
Normal Pulling 20m kubelet Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.1"
Normal Pulled 20m kubelet Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.1"
Normal Created 20m kubelet Created container ingress-azure
Normal Started 20m kubelet Started container ingress-azure
Warning Unhealthy 41s (x117 over 20m) kubelet Readiness probe failed: Get http://10.2.0.83:8123/health/ready: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
kubectl logs -f 显示的日志包含以下错误：
logs error
ERROR: logging before flag.Parse: I1015 07:29:04.152565 1 utils.go:115] Using verbosity level 3 from environment variable APPGW_VERBOSITY_LEVEL
ERROR: logging before flag.Parse: I1015 07:29:04.152726 1 main.go:78] Unable to load cloud provider config '/etc/appgw/azure.json'. Error: Reading Az Context file
"/etc/appgw/azure.json" failed: open /etc/appgw/azure.json: permission denied
E1015 07:29:04.172959 1 context.go:198] Error fetching AGIC Pod (This may happen if AGIC is running in a test environment). Error: pods "ingress-azure-57bcc69687-bqbdn" is forbidden: User "system:serviceaccount:default:ingress-azure" cannot get resource "pods" in API group "" in the namespace "default"
I1015 07:29:04.172990 1 environment.go:240] KUBERNETES_WATCHNAMESPACE is not set. Watching all available namespaces.
I1015 07:29:04.173096 1 main.go:128] Appication Gateway Details: Subscription="e14827fd-ae03-4832-9388-ef0aa3f28693" Resource Group="rg-test" Name="appGateway"
I1015 07:29:04.173107 1 auth.go:46] Creating authorizer from Azure Managed Service Identity
I1015 07:29:04.173365 1 httpserver.go:57] Starting API Server on :8123
I1015 07:33:07.865519 1 main.go:175] Ingress Controller will observe all namespaces.
I1015 07:33:07.894383 1 context.go:132] k8s context run started
I1015 07:33:07.894419 1 context.go:176] Waiting for initial cache sync
E1015 07:33:07.913698 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "ingresses" in API group "extensions" at the cluster scope
E1015 07:33:07.914239 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "services" in API group "" at the cluster scope
E1015 07:33:07.914307 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "secrets" in API group "" at the cluster scope
E1015 07:33:07.914613 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "pods" in API group "" at the cluster scope
E1015 07:33:07.915265 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "ingresses" in API group "extensions" at the cluster scope
E1015 07:33:07.914752 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Endpoints:endpoints is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "endpoints" in API group "" at the cluster scope
E1015 07:33:07.917430 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "services" in API group "" at the cluster scope
E1015 07:33:07.919146 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "secrets" in API group "" at the cluster scope
E1015 07:33:07.919932 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "pods" in API group "" at the cluster scope
E1015 07:33:07.922582 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Endpoints:endpoints is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "endpoints" in API group "" at the cluster scope
E1015 07:33:09.877700 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Endpoints:endpoints is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "endpoints" in API group "" at the cluster scope
E1015 07:33:09.977016 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "services" in API group "" at the cluster scope
E1015 07:33:09.994355 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "secrets" in API group "" at the cluster scope
E1015 07:33:10.030444 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "ingresses" in API group "extensions" at the cluster scope
E1015 07:33:10.612903 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "pods" in API group "" at the cluster scope
E1015 07:33:13.730098 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Endpoints:endpoints is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "endpoints" in API group "" at the cluster scope
E1015 07:33:14.333551 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "services" in API group "" at the cluster scope
E1015 07:33:14.752686 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "pods" in API group "" at the cluster scope
E1015 07:33:15.022569 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "secrets" in API group "" at the cluster scope
E1015 07:33:15.992773 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "ingresses" in API group "extensions" at the cluster scope
E1015 07:33:22.033914 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Endpoints:endpoints is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "endpoints" in API group "" at the cluster scope
E1015 07:33:22.477987 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "pods" in API group "" at the cluster scope
E1015 07:33:25.552073 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "services" in API group "" at the cluster scope
我已经创建了指南中所述的三个角色分配：
- AGIC 的身份 Contributor 访问 App Gateway
- AGIC 的身份Reader 访问 App Gateway 资源组
- 托管身份操作员角色到 AGIC 集群的身份
请帮助我理解错误。
所以我按照这个 blogpost 解决了这个问题。我从之前遵循的指南中更改了两件事：
- 已将 helm-config.yaml 中启用的 rbac 更改为 true
- 使用以下命令安装入口：
helm upgrade --install appgw-ingress-azure -f helm-config.yaml application-gateway-kubernetes-ingress/ingress-azure
虽然 pod 已准备就绪，但在此之后 运行，事件确实表明它不健康。所以就是这样。但是，它解决了之前的问题
我已按照以下指南中的步骤在 azure 中设置魔法： https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/docs/setup/install-existing.md
我有一个 vnet，一个子网中有一个 aks 集群（启用了 rbac），另一个子网中有一个应用程序网关。我已按照使用服务主体和 aad pod 身份授权 ARM 的步骤进行操作。
然而，在这两种情况下，一旦使用 helm-config.yaml 文件安装了入口控制器，pod 的日志就会显示它 运行 但还没有准备好。
以下是使用aad pod身份认证时
kubectl describe pod显示的事件有：
events
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 20m default-scheduler Successfully assigned default/ingress-azure-57bcc69687-bqbdn to aks-agentpool-29530272-vmss000002
Normal Pulling 20m kubelet Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.1"
Normal Pulled 20m kubelet Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.1"
Normal Created 20m kubelet Created container ingress-azure
Normal Started 20m kubelet Started container ingress-azure
Warning Unhealthy 41s (x117 over 20m) kubelet Readiness probe failed: Get http://10.2.0.83:8123/health/ready: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
kubectl logs -f 显示的日志包含以下错误：
logs error
ERROR: logging before flag.Parse: I1015 07:29:04.152565 1 utils.go:115] Using verbosity level 3 from environment variable APPGW_VERBOSITY_LEVEL
ERROR: logging before flag.Parse: I1015 07:29:04.152726 1 main.go:78] Unable to load cloud provider config '/etc/appgw/azure.json'. Error: Reading Az Context file
"/etc/appgw/azure.json" failed: open /etc/appgw/azure.json: permission denied
E1015 07:29:04.172959 1 context.go:198] Error fetching AGIC Pod (This may happen if AGIC is running in a test environment). Error: pods "ingress-azure-57bcc69687-bqbdn" is forbidden: User "system:serviceaccount:default:ingress-azure" cannot get resource "pods" in API group "" in the namespace "default"
I1015 07:29:04.172990 1 environment.go:240] KUBERNETES_WATCHNAMESPACE is not set. Watching all available namespaces.
I1015 07:29:04.173096 1 main.go:128] Appication Gateway Details: Subscription="e14827fd-ae03-4832-9388-ef0aa3f28693" Resource Group="rg-test" Name="appGateway"
I1015 07:29:04.173107 1 auth.go:46] Creating authorizer from Azure Managed Service Identity
I1015 07:29:04.173365 1 httpserver.go:57] Starting API Server on :8123
I1015 07:33:07.865519 1 main.go:175] Ingress Controller will observe all namespaces.
I1015 07:33:07.894383 1 context.go:132] k8s context run started
I1015 07:33:07.894419 1 context.go:176] Waiting for initial cache sync
E1015 07:33:07.913698 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "ingresses" in API group "extensions" at the cluster scope
E1015 07:33:07.914239 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "services" in API group "" at the cluster scope
E1015 07:33:07.914307 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "secrets" in API group "" at the cluster scope
E1015 07:33:07.914613 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "pods" in API group "" at the cluster scope
E1015 07:33:07.915265 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "ingresses" in API group "extensions" at the cluster scope
E1015 07:33:07.914752 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Endpoints:endpoints is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "endpoints" in API group "" at the cluster scope
E1015 07:33:07.917430 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "services" in API group "" at the cluster scope
E1015 07:33:07.919146 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "secrets" in API group "" at the cluster scope
E1015 07:33:07.919932 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "pods" in API group "" at the cluster scope
E1015 07:33:07.922582 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Endpoints:endpoints is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "endpoints" in API group "" at the cluster scope
E1015 07:33:09.877700 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Endpoints:endpoints is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "endpoints" in API group "" at the cluster scope
E1015 07:33:09.977016 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "services" in API group "" at the cluster scope
E1015 07:33:09.994355 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "secrets" in API group "" at the cluster scope
E1015 07:33:10.030444 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "ingresses" in API group "extensions" at the cluster scope
E1015 07:33:10.612903 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "pods" in API group "" at the cluster scope
E1015 07:33:13.730098 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Endpoints:endpoints is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "endpoints" in API group "" at the cluster scope
E1015 07:33:14.333551 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "services" in API group "" at the cluster scope
E1015 07:33:14.752686 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "pods" in API group "" at the cluster scope
E1015 07:33:15.022569 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "secrets" in API group "" at the cluster scope
E1015 07:33:15.992773 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "ingresses" in API group "extensions" at the cluster scope
E1015 07:33:22.033914 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Endpoints:endpoints is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "endpoints" in API group "" at the cluster scope
E1015 07:33:22.477987 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "pods" in API group "" at the cluster scope
E1015 07:33:25.552073 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "services" in API group "" at the cluster scope
我已经创建了指南中所述的三个角色分配：
- AGIC 的身份 Contributor 访问 App Gateway
- AGIC 的身份Reader 访问 App Gateway 资源组
- 托管身份操作员角色到 AGIC 集群的身份
请帮助我理解错误。
所以我按照这个 blogpost 解决了这个问题。我从之前遵循的指南中更改了两件事：
- 已将 helm-config.yaml 中启用的 rbac 更改为 true
- 使用以下命令安装入口：
helm upgrade --install appgw-ingress-azure -f helm-config.yaml application-gateway-kubernetes-ingress/ingress-azure
虽然 pod 已准备就绪，但在此之后 运行，事件确实表明它不健康。所以就是这样。但是，它解决了之前的问题