Kubernetes + Nginx 上的 Jupyterhub - 登录后不生成

Question

Context：我使用 Terraform 在 AWS 上创建了一个 EKS 集群。在那个集群上，我使用 Helm 3 安装了 Nginx Ingress。TLS 是使用 Let's Encrypt with cert-manager 执行的。随后，我可以使用部署、服务和入口 yaml 文件添加 Web 公开的应用程序。

问题：我无法成功部署 JupyterHub。安装和公开工作正常，JupyterHub 使用 TCP 协议，证书管理器成功创建证书。当用户成功登录到 jupyterhub 但当 jupyterhub 应该生成笔记本时出现 invalid or expired cookie token 时，问题就开始了。

问题：我不清楚为什么产卵不起作用以及如何解决这个问题。有没有人有更好地理解这个问题的建议？

jupyterhub_config.py如下：

c = get_config()
c.JupyterHub.authenticator_class = 'jupyterhub.auth.DummyAuthenticator'
c.Authenticator.allowed_users = {'dummy'}
c.Authenticator.admin_users = {'dummy'}
c.DummyAuthenticator.password = "fakenews"
c.JupyterHub.admin_access = True

deployment.yaml如下：

---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  generation: 1
  labels:
    run: jupyterhub
  name: jupyterhub
  namespace: jhub
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      run: jupyterhub
  template:
    metadata:
      creationTimestamp: ~
      labels:
        run: jupyterhub
    spec:
      containers:
        - name: jupyterhub
          image: "jupyterhub/jupyterhub:latest"
          imagePullPolicy: IfNotPresent
          ports:
            -
              containerPort: 8000
              protocol: TCP
          terminationMessagePolicy: File
          volumeMounts:
            -
              mountPath: /srv/jupyterhub/jupyterhub_config.py
              name: jupyterhub-config
              subPath: jupyterhub_config.py
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
        -
          configMap:
            name: jupyterhub-config
          name: jupyterhub-config

ingress.yaml如下：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-resource
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  tls:
  - hosts:
    - hub.example.com
    secretName: hub-example-com-tls
  rules:
  - host: hub.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: jupyterhub
          servicePort: 8000

使用的命令：

$ kubectl create configmap jupyterhub-config --from-file=./jupyterhub_config.py
$ kubectl create -f deployment.yaml
$ kubectl expose deployment jupyterhub
$ kubectl apply -f ingress.yaml

这导致在 https://hub.example.com 上成功安全部署 Web 服务。但是登录后，jupyterhub 容器日志在尝试生成 jupyter 实例时给出 invalid or expired cookie token。

[I 2020-08-21 08:26:42.725 JupyterHub app:2307] Running JupyterHub version 1.2.0dev
[I 2020-08-21 08:26:42.726 JupyterHub app:2338] Using Authenticator: jupyterhub.auth.DummyAuthenticator-1.2.0dev
[I 2020-08-21 08:26:42.726 JupyterHub app:2338] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-1.2.0dev
[I 2020-08-21 08:26:42.726 JupyterHub app:2338] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.2.0dev
[I 2020-08-21 08:26:42.735 JupyterHub app:1442] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret
[I 2020-08-21 08:26:42.752 alembic.runtime.migration migration:155] Context impl SQLiteImpl.
[I 2020-08-21 08:26:42.752 alembic.runtime.migration migration:162] Will assume non-transactional DDL.
[I 2020-08-21 08:26:42.758 alembic.runtime.migration migration:515] Running stamp_revision  -> 4dc2d5a8c53c
[I 2020-08-21 08:26:42.809 JupyterHub proxy:461] Generating new CONFIGPROXY_AUTH_TOKEN
[I 2020-08-21 08:26:42.850 JupyterHub app:2377] Initialized 0 spawners in 0.002 seconds
[W 2020-08-21 08:26:42.853 JupyterHub proxy:643] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2020-08-21 08:26:42.853 JupyterHub proxy:646] Starting proxy @ http://:8000
08:26:43.359 [ConfigProxy] info: Proxying http://*:8000 to (no default)
08:26:43.362 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
08:26:43.474 [ConfigProxy] info: 200 GET /api/routes 
[I 2020-08-21 08:26:43.475 JupyterHub app:2622] Hub API listening on http://127.0.0.1:8081/hub/
08:26:43.476 [ConfigProxy] info: 200 GET /api/routes 
[I 2020-08-21 08:26:43.476 JupyterHub proxy:320] Checking routes
[I 2020-08-21 08:26:43.476 JupyterHub proxy:400] Adding default route for Hub: / => http://127.0.0.1:8081
08:26:43.478 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
08:26:43.478 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
08:26:43.478 [ConfigProxy] info: 201 POST /api/routes/ 
[I 2020-08-21 08:26:43.479 JupyterHub app:2697] JupyterHub is now running at http://:8000
[I 2020-08-21 08:26:56.023 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@10.0.1.148) 1.16ms
[I 2020-08-21 08:27:01.409 JupyterHub base:742] User logged in: dummy
[I 2020-08-21 08:27:01.429 JupyterHub log:181] 302 POST /hub/login?next= -> /hub/spawn (dummy@10.0.1.148) 68.74ms
[I 2020-08-21 08:27:01.758 JupyterHub log:181] 200 GET /hub/login?next=%2Fhub%2Fspawn (@10.0.1.148) 219.05ms
08:31:43.482 [ConfigProxy] info: 200 GET /api/routes 
[I 2020-08-21 08:31:43.482 JupyterHub proxy:320] Checking routes
[I 2020-08-21 12:06:43.482 JupyterHub proxy:320] Checking routes
[I 2020-08-21 12:07:08.386 JupyterHub log:181] 200 GET /hub/login?next=%2Fhub%2Fspawn (@10.0.2.117) 1.85ms
[I 2020-08-21 12:07:13.216 JupyterHub base:742] User logged in: dummy
[I 2020-08-21 12:07:13.217 JupyterHub log:181] 302 POST /hub/login?next=%2Fhub%2Fspawn -> /hub/spawn (dummy@10.0.2.117) 5.40ms
[I 2020-08-21 12:07:13.309 JupyterHub log:181] 200 GET /hub/login?next=%2Fhub%2Fspawn (@10.0.2.117) 1.22ms
[I 2020-08-21 13:27:28.324 JupyterHub log:181] 302 GET / -> /hub/ (@10.0.2.117) 0.90ms 
[I 2020-08-21 13:27:28.410 JupyterHub log:181] 200 GET /hub/login (@10.0.2.117) 1.28ms 
[W 2020-08-21 13:27:34.613 JupyterHub base:392] Invalid or expired cookie token 
[I 2020-08-21 13:27:34.615 JupyterHub log:181] 302 GET /hub/spawn -> /hub/login?next=%2Fhub%2Fspawn (@10.0.2.117) 1.88ms

Answer 1

将副本数从 2 更改为 1 可以解决此问题。谢谢！非常不幸的是，它不适用于副本。

Answer 2

正如 OP 所提到的，将部署缩减到 1 个副本解决了这个问题。

我想澄清一下似乎是什么问题。

Jupyterhub 不可扩展。它是有状态应用程序，并且（截至目前）不可能使其运行高度可用。

K8s 服务在两个 pods/replicas 之间进行负载均衡，随机发送流量。

登录到一个 jupyterhub 后，您会收到一个令牌。现在使用此令牌发送另一个请求。如果这个带有您刚刚收到的令牌的请求被发送到 jupyterhub 的第二个实例，您能猜到会发生什么吗？不知道这个令牌是什么的人，因为它不是生成它的人。

invalid or expired cookie token

这就是您所看到的。第二个实例会发现此令牌无效。

这就是缩小到一个副本可以解决问题的原因。这是因为现在所有流量都发送到一个 pod。

Kubernetes + Nginx 上的 Jupyterhub - 登录后不生成

Jupyterhub on Kubernetes + Nginx - No spawning after login

jupyterhub