K8s Ingress,启动ingress controller nginx报错?

K8s Ingress, initiate ingress controller nginx error?

我有两个 spring 引导容器,我想设置入口服务。正如文档here所说,ingress有两部分,一个是controller,一个是resources。

我的两个资源是两个容器:gearbox-rack-eureka-server和gearbox-rack-config-server。不同之处在于端口,因此入口可以通过不同的端口路由流量。我的 yaml 文件如下:

eureka_pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: gearbox-rack-eureka-server
  labels:
    app: gearbox-rack-eureka-server
    purpose: platform_eureka_demo
spec:
  containers:
  - name:  gearbox-rack-eureka-server
    image: 192.168.1.229:5000/gearboxrack/gearbox-rack-eureka-server
    ports:
        - containerPort: 8761

eureka_svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: gearbox-rack-eureka-server
  labels:
    name: gearbox_rack_eureka_server
spec:
  selector:
    app: gearbox-rack-eureka-server
  type: NodePort
  ports:
    - port: 8761
      nodePort: 31501
      name: tcp

config_pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: gearbox-rack-config-server
  labels:
    app: gearbox-rack-config-server
    purpose: platform-demo
spec:
  containers:
  - name:  gearbox-rack-config-server
    image: 192.168.1.229:5000/gearboxrack/gearbox-rack-config-server
    ports:
    - containerPort: 8888
    env:
      - name: EUREKA_SERVER
        value: http://172.16.100.83:8761

config_svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: gearbox-rack-config-server
  labels:
    name: gearbox-rack-config-server
spec:
  selector:
    app: gearbox-rack-config-server
  type: NodePort
  ports:
    - port: 8888
      nodePort: 31502
      name: tcp

我的 ingress-nginx 控制器大部分是复制的 from the link above

ingress_nginx_ctl.yaml:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
spec:
  type: LoadBalancer
  selector:
    app: ingress-nginx
  ports:
  - name: http
    port: 80
    targetPort: http
  - name: https
    port: 443
    targetPort: https
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: ingress-nginx
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: ingress-nginx
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - image: nginx:1.13.12
        name: ingress-nginx
        imagePullPolicy: Always
        ports:
          - name: http
            containerPort: 80
            protocol: TCP
          - name: https
            containerPort: 443
            protocol: TCP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/nginx-default-backend

我执行了以下命令,它们成功了。

kubectl apply -f eureka_pod.yaml
kubectl apply -f eureka_svc.yaml
kubectl apply -f config_pod.yaml
kubectl apply -f config_svc.yaml

然后我在执行 kubectl apply -f ingress_nginx_ctl.yaml 时出错,pod 没有启动,日志如下:

[root@master3 nginx-ingress-controller]# kubectl get pods
NAME                             READY     STATUS             RESTARTS   AGE
gearbox-rack-config-server       1/1       Running            0          39m
gearbox-rack-eureka-server       1/1       Running            0          40m
ingress-nginx-686c9975d5-7d464   0/1       CrashLoopBackOff   6          7m
[root@master3 nginx-ingress-controller]# kubectl logs -f ingress-nginx-686c9975d5-7d464
container_linux.go:247: starting container process caused "exec: \"/nginx-ingress-controller\": stat /nginx-ingress-controller: no such file or directory"

我在root下新建了一个目录/nginx-ingress-controller,再重复一遍,还是报同样的错误。有人可以指出我的问题吗?

我把我的ingress_nginx_res.yaml如下,供参考,可能也有错误。

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - host: 172.16.100.83
    http:
      paths:
      - backend:
          serviceName: gearbox-rack-eureka-server
          servicePort: 8761
  - host: 172.16.100.83
    http:
      paths:
      - path:
        backend:
          serviceName: gearbox-rack-config-server
          servicePort: 8888

==========================================

第二版

换图后link,之前的错误消失了,但仍然存在以下权限问题:

[root@master3 ingress]# kubectl get pods
NAME                             READY     STATUS             RESTARTS   AGE
gearbox-rack-config-server       1/1       Running            0          15m
gearbox-rack-eureka-server       1/1       Running            0          15m
ingress-nginx-8679f9c8ff-5sxw7   0/1       CrashLoopBackOff   5          12m

日志信息如下:

[root@master3 kube]# kubectl logs ingress-nginx-8679f9c8ff-5sxw7
W0530 07:54:22.290114       5 client_config.go:533] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0530 07:54:22.290374       5 main.go:158] Creating API client for https://10.96.0.1:443
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.15.0
  Build:      git-df61bd7
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------
I0530 07:54:22.298248       5 main.go:202] Running in Kubernetes Cluster version v1.9 (v1.9.2) - git (clean) commit 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b - platform linux/amd64
F0530 07:54:22.298610       5 main.go:80] ✖ It seems the cluster it is running with Authorization enabled (like RBAC) and there is no permissions for the ingress controller. Please check the configuration

是RBAC问题。我检查了从论坛下载的安装脚本:

heapster-rbac.yaml:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: heapster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:heapster
subjects:
- kind: ServiceAccount
  name: heapster
  namespace: kube-system

其中一个相关的kubelet启动参数如下:(不知是否相关)。

Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"

通过什么方式,我可以授予入口控制器权限?只需将命名空间 kube-system 放入 ingress_nginx_ctl.yaml?

============================================= ===================

第三版

我把李坤的代码写成ingress_nginx_role_rb.yaml,运行下面的命令:

kubectl apply -f eureka_pod.yaml
kubectl apply -f eureka_svc.yaml
kubectl apply -f config_pod.yaml
kubectl apply -f config_svc.yaml
kubectl apply -f ingress_nginx_role_rb.yaml (just copy paste from Kun Li's answer)
kubectl apply -f nginx_default_backend.yaml
kubectl apply -f ingress_nginx_ctl.yaml

nginx_default_backend.yaml 文件如下:

kind: Service
apiVersion: v1
metadata:
  name: nginx-default-backend
  namespace: kube-system
spec:
  ports:
  - port: 80
    targetPort: http
  selector:
    app: nginx-default-backend
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: nginx-default-backend
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx-default-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        image: chenliujin/defaultbackend
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP

ingress_nginx_ctl.yaml如下:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
spec:
  type: LoadBalancer
  selector:
    app: ingress-nginx
  ports:
  - name: http
    port: 80
    targetPort: http
  - name: https
    port: 443
    targetPort: https
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: ingress-nginx
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: ingress-nginx
    spec:
      terminationGracePeriodSeconds: 60
      serviceAccount: lb
      containers:
      - image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.15.0
        name: ingress-nginx
        imagePullPolicy: Always
        ports:
          - name: http
            containerPort: 80
            protocol: TCP
          - name: https
            containerPort: 443
            protocol: TCP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/nginx-default-backend

从这里,我们可以看到服务 ingress-nginx 命名空间是默认的,而不是 kube-system。但无论如何,控制器启动了。

[root@master3 ingress]# kubectl get pods -n kube-system
NAME                                      READY     STATUS    RESTARTS   AGE
calico-etcd-cdn8z                         1/1       Running   0          11m
calico-kube-controllers-d554689d5-tzdq5   1/1       Running   0          11m
calico-node-dz4d6                         2/2       Running   1          11m
coredns-65dcdb4cf-h62bh                   1/1       Running   0          11m
etcd-master3                              1/1       Running   0          10m
heapster-5c448886d-swp58                  1/1       Running   0          11m
ingress-nginx-6ccc799fbc-hq2rm            1/1       Running   0          9m
kube-apiserver-master3                    1/1       Running   0          10m

ingress-nginx pod 的命名空间是 kube-system(如上所示),但其服务的命名空间是默认的。(如下所示)。

[root@master3 ingress]# kubectl get service
NAME                         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
gearbox-rack-config-server   NodePort       10.97.211.136   <none>        8888:31502/TCP               43m
gearbox-rack-eureka-server   NodePort       10.106.69.13    <none>        8761:31501/TCP               43m
ingress-nginx                LoadBalancer   10.105.114.64   <pending>     80:30646/TCP,443:31332/TCP   42m
kubernetes                   ClusterIP      10.96.0.1       <none>        443/TCP                      44m

如评论所述,专家的回复帮助我前进。

对于ingress-controller,应该使用image quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.15.0。你需要设置 nginx-default-backend pod 和服务。

关于 RBAC,我认为您需要一个服务帐户来部署您的 nginx-ingress-controller,具有以下角色和绑定:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: lb
  namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-normal
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-minimal
  namespace: kube-system
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      - "ingress-controller-leader-dev"
      - "ingress-controller-leader-prod"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-minimal
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-minimal
subjects:
  - kind: ServiceAccount
    name: lb
    namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-normal
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-normal
subjects:
  - kind: ServiceAccount
    name: lb
    namespace: kube-system