Azure Kubernetes 集群上的 Websocket 实现错误无法正常工作

Question

我正在尝试使 websocket 服务在我们组织环境中的 Azure Kubernetes 集群上运行。我现有的环境也有 REST api 和 Angular 应用程序使用 ssl 处理入口。但是当我在入口上添加 websocket 服务时，它不起作用。

因此，我尝试使用 Azure 免费订阅首先实现相同的 WITHOUT SSL。对于我的应用程序，我启用了 Http 路由并使用了注释 addon-http-application-routing。

我遇到了以下错误。 'ws://40.119.7.246/ws' 失败：WebSocket 握手期间出错：意外的响应代码：404

请帮助验证我哪里做错了？

配置详情如下。

Dockerfile

FROM node:alpine
WORKDIR /app
COPY package*.json /app/
RUN npm install

COPY ./ /app/
RUN npm run build

CMD ["node","./dist/server.js"]

EXPOSE 8010

socketserver.yaml - Contains Demployment & Service.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: socketserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: socketserver
  template:
    metadata:
      labels:
        app: socketserver
    spec:
      containers:
      - name: socketserver
        image: regkompella.azurecr.io/socketserver:1.0.0
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 8010
      imagePullSecrets:
      - name: regkompella-azurecr-io
---
apiVersion: v1
kind: Service
metadata:
  name: socketserver-svc
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8010
  selector:
    app: socketserver
  type: ClusterIP
---

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: demo-ingress
  annotations:
    kubernetes.io/ingress.class: addon-http-application-routing
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-body-size: 10m
    nginx.ingress.kubernetes.io/websocket-services: socketserver-svc
    nginx.org/websocket-services: socketserver-svc
spec:
  rules:
  - host: demosocket.com
  - http:
      paths:
      - path: /
        backend:
          serviceName: angular-application-svc
          servicePort: 80
      - path: /ws
        backend:
          serviceName: socketserver-svc
          servicePort: 80

Answer 1

阅读了大量文章并参考了一些 github 论坛（在下面添加了参考文章）。在完成这两件事之后，我的 websocket 实现开始工作。我不确定这是否是正确的方法。我完全通过跟踪和错误方法实现了这个解决方案。因此，我请各位有把握的人，如果有更好的方法来解决我的问题，请提出建议。总是对我的步骤持保留态度。

从 link 安装了 NGINX Ingress 控制器。

因为我使用的是 Azure Kubernetes 服务，所以我应用了文档中的以下 yaml。

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud/deploy.yaml

接下来我对我的服务的演示入口配置进行了必要的更改。

我了解到 kubernetes.io/ingress.class: addon-http-application-routing 注释不支持 websocketing。所以，不得不禁用它。

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: demo-ingress
  annotations:
    # this one annotation is making the websocket work.
    nginx.ingress.kubernetes.io/websocket-services: socketserver-svc

    # this one I left as-is. And not playing any role for this websocket 
    # implementation to work
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"

    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-body-size: 10m

    # I thought sticky session is also required for websocket to work. But seems 
    # this has no effect after I installed nginx ingress controller.
    # so disabled all the below annotations also.

    #nginx.org/websocket-services: socketserver-svc
    #nginx.ingress.kubernetes.io/affinity: cookie
    #nginx.ingress.kubernetes.io/affinity-mode: balanced
    #nginx.ingress.kubernetes.io/session-cookie-samesite: Strict
    #kubernetes.io/ingress.class: nginx
    #kubernetes.io/ingress.class: addon-http-application-routing

spec:
  rules:
  - host: demosocket.com
  - http:
      paths:
      - path: /ws
        backend:
          serviceName: socketserver-svc
          servicePort: 80

我试图通过 public IP 地址访问。并且我可以成功收发消息。

ws://52.188.38.118/ws

现在，如果我想在不安装 NGINX Ingress Controller（在步骤 1 中指示）的情况下使 websocket 实现工作，并且想尝试使用 AKS/minikube 附带的默认入口控制器怎么办？答案如下。

从上面的步骤，

a) 避免第 1 步：安装 NGINX Ingress Controller。

b) 只有在入口处需要进行的更改如下。在 ingress yaml 文件上使用以下注释代替步骤 2 中指示的注释。一切都会开始工作。

# this annotation is making my web application also work if I plan to configure something in future.
nginx.ingress.kubernetes.io/ingress.class: nginx

# this one annotation is making the websocket work.
nginx.ingress.kubernetes.io/websocket-services: socketserver-svc

# by default ssl is true - as I am trying locally and want to disable ssl-# redirect. So set this to false.
nginx.ingress.kubernetes.io/ssl-redirect: "false"

# Below are just additional annotation to allow CORS etc.
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"
nginx.ingress.kubernetes.io/proxy-body-size: 10m

参考文章：

https://medium.com/flant-com/comparing-ingress-controllers-for-kubernetes-9b397483b46b
https://kubernetes.github.io/ingress-nginx/deploy/#azure
先生。 dstrebel 的评论 -> https://github.com/Azure/AKS/issues/768

I typically recommend just setting up a Ingress Controller on the cluster and not enabling "http-application-routing", as there's a lot of limitations to it. The goal with HTTP Application ROuting was for users to get setup quickly with Ingress, but not really for production deployments due to the limitations of the configuration.
DenisBiondic 于 2018 年 10 月 2 日发表评论 -> https://github.com/Azure/AKS/issues/672

I am not 100% certain, since I don't use application routing feature, but >I think it does not use the https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/websocket controller but rather the https://github.com/kubernetes/ingress-nginx. In case of the latter, I think enabling session affinity with cookies might be enough: https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md#session-affinity

In your case you are using the wrong annotation which does not work with application routing ingress controller under the hood.

我欢迎提出建议和最佳做法。

Azure Kubernetes 集群上的 Websocket 实现错误无法正常工作

Error on Websocket Implementation on Azure Kubernetes Cluster Not working

websocket

kubernetes-ingress

azure-aks

nginx-ingress