Kubernetes nginx ingress + oauth2 外部认证超时

Question

我正在尝试使用 Azure AD 作为外部身份验证提供程序，使用 oauth2_proxy 保护服务的状态页面。目前，如果我浏览到应用程序 (https://sub.domain.com/service/hangfire) 的 public url，我会收到 504 网关超时，它应该指示我进行身份验证。

我主要是按照这个指南作为参考：https://msazure.club/protect-kubernetes-webapps-with-azure-active-directory-aad-authentication/

如果我禁用指导身份验证的注释，我可以毫无问题地进入 public 状态页面。如果我浏览到 https://sub.domain.com/oauth2，我会收到一个提示，要求我向我的提供商进行身份验证，这是我所期望的。我不确定问题出在入口配置中，但我无法在网上、Whosebug 或其他地方找到任何类似的案例。

在这种情况下，所有内容（oauth 部署、服务和入口规则）都位于 'dev' 命名空间中，但实际入口部署除外，它位于自己的命名空间中。我不怀疑这有什么不同，但 SSL 终止是由集群外部的网关处理的。

oauth2 部署：

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: oauth2-proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: oauth2-proxy
  template:
    metadata:
      labels:
        app: oauth2-proxy
    spec:
      containers:
      - name: oauth2-proxy
        image: quay.io/pusher/oauth2_proxy:v3.2.0
        imagePullPolicy: IfNotPresent
        args:
        - --provider=azure
        - --email-domain=domain.com
        - --upstream=http://servicename
        - --http-address=0.0.0.0:4180
        - --azure-tenant=id
        - --client-id=id
        - --client-secret=number
        env:
         - name: OAUTH2_PROXY_COOKIE_SECRET
           value: secret
        ports:
         - containerPort: 4180
           protocol : TCP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: oauth2-proxy
  name: oauth2-proxy
spec:
  ports:
  - name: http
    port: 4180
    protocol: TCP
    targetPort: 4180
  selector:
    app: oauth2-proxy

入口规则：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: service-ingress1
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/auth-url: https://sub.domain.com/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: https://sub.domain.com/oauth2/start?rd=$https://sub.domain.com/service/hangfire"
spec:
  rules:
  - host: sub.domain.com       
    http:
      paths:
      - path: /service/hangfire
        backend:
          serviceName: service
          servicePort: 80                    
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: service-oauth2-proxy
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: sub.domain.com      
    http:
      paths:
      - path: /oauth2
        backend:
          serviceName: oauth2-proxy
          servicePort: 4180

当我浏览到 url 时出现 504 错误，但我在入口 pods.

中没有看到任何错误

Answer 1

这就是我对 Azure AD 的 oAuth 代理所做的事情：

  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"

我一直在使用这个 oAuth 代理：

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: oauth2-proxy
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: oauth2-proxy
  template:
    metadata:
      labels:
        app: oauth2-proxy
    spec:
      containers:
      - env:
          - name: OAUTH2_PROXY_PROVIDER
            value: azure
          - name: OAUTH2_PROXY_AZURE_TENANT
            value: xxx
          - name: OAUTH2_PROXY_CLIENT_ID
            value: yyy
          - name: OAUTH2_PROXY_CLIENT_SECRET
            value: zzz
          - name: OAUTH2_PROXY_COOKIE_SECRET
            value: anyrandomstring
          - name: OAUTH2_PROXY_HTTP_ADDRESS
            value: "0.0.0.0:4180"
          - name: OAUTH2_PROXY_UPSTREAM
            value: "http://where_to_redirect_to:443"
        image: machinedata/oauth2_proxy:latest
        imagePullPolicy: IfNotPresent
        name: oauth2-proxy
        ports:
        - containerPort: 4180
          protocol: TCP

Answer 2

我的设置类似于 4c74356b41 的

oauth2-代理部署

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: oauth2-proxy
  name: oauth2-proxy
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: oauth2-proxy
  template:
    metadata:
      labels:
        app: oauth2-proxy
    spec:
      containers:
      - args:
        - --azure-tenant=TENANT-GUID
        - --email-domain=company.com
        - --http-address=0.0.0.0:4180
        - --provider=azure
        - --upstream=file:///dev/null
        env:
        - name: OAUTH2_PROXY_CLIENT_ID
          valueFrom:
            secretKeyRef:
              key: client-id
              name: oauth2-proxy
        - name: OAUTH2_PROXY_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              key: client-secret
              name: oauth2-proxy
        - name: OAUTH2_PROXY_COOKIE_SECRET
          valueFrom:
            secretKeyRef:
              key: cookie-secret
              name: oauth2-proxy
        image: quay.io/pusher/oauth2_proxy:v3.1.0
        name: oauth2-proxy

oauth2-代理服务

apiVersion: v1
kind: Service
metadata:
  labels:
    app: oauth2-proxy
  name: oauth2-proxy
  namespace: monitoring
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app: oauth2-proxy
  type: ClusterIP

oauth2-代理入口

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  labels:
    app: oauth2-proxy
  name: oauth2-proxy
  namespace: monitoring
spec:
  rules:
  - host: myapp.hostname.net
    http:
      paths:
      - backend:
          serviceName: oauth2-proxy
          servicePort: 80
        path: /oauth2

oauth2-代理配置

apiVersion: v1
kind: Secret
metadata:
  labels:
    app: oauth2-proxy
  name: oauth2-proxy
  namespace: monitoring
data:
# Values below are fake
  client-id: AAD_CLIENT_ID
  client-secret: AAD_CLIENT_SECRET
  cookie-secret: COOKIE_SECRET

使用 AAD Ingress 的应用程序

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/auth-signin: https://$host/oauth2/start?rd=$request_uri
    nginx.ingress.kubernetes.io/auth-url: https://$host/oauth2/auth
  labels:
    app: myapp
  name: myapp
  namespace: monitoring
spec:
  rules:
  - host: myapp.hostname.net
    http:
      paths:
      - backend:
          serviceName: myapp
          servicePort: 80
        path: /
  tls:
  - hosts:
    - myapp.hostname.net

需要完成的另一个步骤是将重定向 URI 添加到 AAD 应用程序注册中。在 Azure 门户中导航到您的 AAD 应用程序注册 > 身份验证 > 添加 https://myapp.hostname.net/oauth2/callback 到重定向 URI > 保存

Answer 3

我最终在这里找到了解决方案：https://github.com/helm/charts/issues/5958

我不得不使用 auth-url 的内部服务地址，我在其他任何地方都没有提到过。

nginx.ingress.kubernetes.io/auth-url: http://oauth2-proxy.development.svc.cluster.local:4180/oauth2/auth

Kubernetes nginx ingress + oauth2 外部认证超时

Kubernetes nginx ingress + oauth2 external auth timing out

kubernetes

oauth-2.0

azure-active-directory

azure-aks

nginx-ingress