postgresql bitnami helm chart 上的 CrashLoopBackOff

CrashLoopBackOff on postgresql bitnami helm chart

我知道已经有很多关于这个的问题,我已经阅读了其中的大部分,但我的问题似乎不适合他们。

我正在 运行使用 helm 图表从 bitnami 中安装一个 postgresql,如下所述。干净的设置没问题,一切都开始正常。但是过了一段时间,直到现在我都找不到任何模式,pod 进入 CrashLoopBackOff 状态,无论我尝试什么都无法恢复它!

Helm uninstall/install 没有解决问题。 PV似乎是问题所在,但我不知道为什么。 而且我没有收到任何错误消息,这是其中奇怪和可怕的部分。

我使用 minikube 运行 k8s 和 helm v3。

以下是定义和日志:

# Source: aposphere/charts/sessiondb/templates/svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: sessiondb
  labels:
    app.kubernetes.io/name: sessiondb
    helm.sh/chart: sessiondb-10.9.6
    app.kubernetes.io/instance: asdf
    app.kubernetes.io/managed-by: Helm
  annotations:
  namespace: default
spec:
  type: ClusterIP
  ports:
    - name: tcp-postgresql
      port: 5432
      targetPort: tcp-postgresql
  selector:
    app.kubernetes.io/name: sessiondb
    app.kubernetes.io/instance: asdf
    role: primary
---
# Source: aposphere/charts/sessiondb/templates/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: sessiondb
  labels:
    app.kubernetes.io/name: sessiondb
    helm.sh/chart: sessiondb-10.9.6
    app.kubernetes.io/instance: asdf
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: primary
  annotations:
  namespace: default
spec:
  serviceName: sessiondb-headless
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/name: sessiondb
      app.kubernetes.io/instance: asdf
      role: primary
  template:
    metadata:
      name: sessiondb
      labels:
        app.kubernetes.io/name: sessiondb
        helm.sh/chart: sessiondb-10.9.6
        app.kubernetes.io/instance: asdf
        app.kubernetes.io/managed-by: Helm
        role: primary
        app.kubernetes.io/component: primary
    spec:      
      affinity:
        podAffinity:
          
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: sessiondb
                    app.kubernetes.io/instance: asdf
                    app.kubernetes.io/component: primary
                namespaces:
                  - "default"
                topologyKey: kubernetes.io/hostname
              weight: 1
        nodeAffinity:
          
      securityContext:
        fsGroup: 1001
      automountServiceAccountToken: false
      containers:
        - name: sessiondb
          image: docker.io/bitnami/postgresql:11.13.0-debian-10-r33
          imagePullPolicy: "IfNotPresent"
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
          securityContext:
            runAsUser: 1001
          env:
            - name: BITNAMI_DEBUG
              value: "false"
            - name: POSTGRESQL_PORT_NUMBER
              value: "5432"
            - name: POSTGRESQL_VOLUME_DIR
              value: "/bitnami/postgresql"
            - name: PGDATA
              value: "/bitnami/postgresql/data"
            - name: POSTGRES_USER
              value: "postgres"
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgresql-root-password
                  key: postgresql-password
            - name: POSTGRES_DB
              value: "session"
            - name: POSTGRESQL_ENABLE_LDAP
              value: "no"
            - name: POSTGRESQL_ENABLE_TLS
              value: "no"
            - name: POSTGRESQL_LOG_HOSTNAME
              value: "false"
            - name: POSTGRESQL_LOG_CONNECTIONS
              value: "false"
            - name: POSTGRESQL_LOG_DISCONNECTIONS
              value: "false"
            - name: POSTGRESQL_PGAUDIT_LOG_CATALOG
              value: "off"
            - name: POSTGRESQL_CLIENT_MIN_MESSAGES
              value: "error"
            - name: POSTGRESQL_SHARED_PRELOAD_LIBRARIES
              value: "pgaudit"
          ports:
            - name: tcp-postgresql
              containerPort: 5432
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - exec pg_isready -U "postgres" -d "dbname=session" -h 127.0.0.1 -p 5432
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 6
          readinessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - -e
                - |
                  exec pg_isready -U "postgres" -d "dbname=session" -h 127.0.0.1 -p 5432
                  [ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ]
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 6
          volumeMounts:
            - name: custom-init-scripts
              mountPath: /docker-entrypoint-initdb.d/
            - name: dshm
              mountPath: /dev/shm
            - name: data
              mountPath: /bitnami/postgresql
              subPath: 
      volumes:
        - name: custom-init-scripts
          configMap:
            name: sessiondb-scheme
        - name: dshm
          emptyDir:
            medium: Memory
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: "8Gi"

容器日志:

% kubectl logs sessiondb-0
postgresql 10:09:01.48 
postgresql 10:09:01.48 Welcome to the Bitnami postgresql container
postgresql 10:09:01.49 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql
postgresql 10:09:01.49 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues
postgresql 10:09:01.49 
postgresql 10:09:01.50 DEBUG ==> Configuring libnss_wrapper...
postgresql 10:09:01.51 INFO  ==> ** Starting PostgreSQL setup **
postgresql 10:09:01.54 INFO  ==> Validating settings in POSTGRESQL_* env vars..
postgresql 10:09:01.55 INFO  ==> Loading custom pre-init scripts...
postgresql 10:09:01.55 INFO  ==> Initializing PostgreSQL database...
postgresql 10:09:01.56 DEBUG ==> Ensuring expected directories/files exist...
postgresql 10:09:01.57 INFO  ==> pg_hba.conf file not detected. Generating it...
postgresql 10:09:01.58 INFO  ==> Generating local authentication configuration
postgresql 10:09:01.58 INFO  ==> Deploying PostgreSQL with persisted data...
postgresql 10:09:01.60 INFO  ==> Configuring replication parameters
postgresql 10:09:01.65 INFO  ==> Configuring fsync
postgresql 10:09:01.71 INFO  ==> Loading custom scripts...
postgresql 10:09:01.72 INFO  ==> Loading user's custom files from /docker-entrypoint-initdb.d ...
postgresql 10:09:01.72 INFO  ==> Starting PostgreSQL in background...
pg_ctl: directory "/bitnami/postgresql/data" is not a database cluster directory

之后容器终止,不再有日志!

初始化容器的日志:

% kubectl logs sessiondb-0 init-chmod-data
+ chown 1001:1001 /bitnami/postgresql
+ mkdir -p /bitnami/postgresql/data
+ chmod 700 /bitnami/postgresql/data
+ find /bitnami/postgresql -mindepth 1 -maxdepth 1 -not -name conf -not -name .snapshot -not -name lost+found
+ xargs chown -R 1001:1001
+ chmod -R 777 /dev/shm

权限:

I have no name!@sessiondb-0:/$ stat /bitnami/postgresql/data
  File: /bitnami/postgresql/data
  Size: 207         Blocks: 0          IO Block: 4096   directory
Device: 10301h/66305d   Inode: 712929      Links: 12
Access: (0700/drwx------)  Uid: ( 1001/ UNKNOWN)   Gid: ( 1001/ UNKNOWN)
Access: 2021-11-10 15:16:13.958633094 +0000
Modify: 2021-11-26 08:40:42.621884636 +0000
Change: 2021-11-26 10:37:47.844490933 +0000
 Birth: -

描述资源:

Name:         sessiondb-0
Namespace:    default
Priority:     0
Node:         ip-10-0-1-112.eu-central-1.compute.internal/10.0.1.112
Start Time:   Fri, 26 Nov 2021 10:40:02 +0100
Labels:       app.kubernetes.io/component=primary
              app.kubernetes.io/instance=asdf
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=sessiondb
              controller-revision-hash=sessiondb-578ddf476b
              helm.sh/chart=sessiondb-10.9.6
              role=primary
              statefulset.kubernetes.io/pod-name=sessiondb-0
Annotations:  <none>
Status:       Running
IP:           172.17.0.4
IPs:
  IP:           172.17.0.4
Controlled By:  StatefulSet/sessiondb
Containers:
  sessiondb:
    Container ID:   docker://a94f894687f0813196a94afe88f64723f238eb7d2cb061e4c7ef17354f27dee8
    Image:          docker.io/bitnami/postgresql:11.13.0-debian-10-r33
    Image ID:       docker-pullable://bitnami/postgresql@sha256:205e1c5a1d4b56d0d63f6579557652f958e321006c4cb5325d031d40313e4ea2
    Port:           5432/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 26 Nov 2021 10:50:45 +0100
      Finished:     Fri, 26 Nov 2021 10:50:46 +0100
    Ready:          False
    Restart Count:  7
    Requests:
      cpu:      250m
      memory:   256Mi
    Liveness:   exec [/bin/sh -c exec pg_isready -U "postgres" -d "dbname=session" -h 127.0.0.1 -p 5432] delay=30s timeout=5s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/sh -c -e exec pg_isready -U "postgres" -d "dbname=session" -h 127.0.0.1 -p 5432
[ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ]
] delay=5s timeout=5s period=10s #success=1 #failure=6
    Environment:
      BITNAMI_DEBUG:                        false
      POSTGRESQL_PORT_NUMBER:               5432
      POSTGRESQL_VOLUME_DIR:                /bitnami/postgresql
      PGDATA:                               /bitnami/postgresql/data
      POSTGRES_USER:                        postgres
      POSTGRES_PASSWORD:                    <set to the key 'postgresql-password' in secret 'postgresql-root-password'>  Optional: false
      POSTGRES_DB:                          session
      POSTGRESQL_ENABLE_LDAP:               no
      POSTGRESQL_ENABLE_TLS:                no
      POSTGRESQL_LOG_HOSTNAME:              false
      POSTGRESQL_LOG_CONNECTIONS:           false
      POSTGRESQL_LOG_DISCONNECTIONS:        false
      POSTGRESQL_PGAUDIT_LOG_CATALOG:       off
      POSTGRESQL_CLIENT_MIN_MESSAGES:       error
      POSTGRESQL_SHARED_PRELOAD_LIBRARIES:  pgaudit
    Mounts:
      /bitnami/postgresql from data (rw)
      /dev/shm from dshm (rw)
      /docker-entrypoint-initdb.d/ from custom-init-scripts (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-sessiondb-0
    ReadOnly:   false
  custom-init-scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sessiondb-scheme
    Optional:  false
  dshm:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      Memory
    SizeLimit:   <unset>
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  11m                  default-scheduler  Successfully assigned default/sessiondb-0 to ip-10-0-1-112.eu-central-1.compute.internal
  Normal   Created    11m (x4 over 11m)    kubelet            Created container sessiondb
  Normal   Started    11m (x4 over 11m)    kubelet            Started container sessiondb
  Normal   Pulled     10m (x5 over 11m)    kubelet            Container image "docker.io/bitnami/postgresql:11.13.0-debian-10-r33" already present on machine
  Warning  BackOff    110s (x57 over 11m)  kubelet            Back-off restarting failed container

---

% kubectl describe pvc data-sessiondb-0
Name:          data-sessiondb-0
Namespace:     default
StorageClass:  standard
Status:        Bound
Volume:        pvc-6b56b20c-3e56-4a92-9278-794bf6cda4de
Labels:        app.kubernetes.io/instance=asdf
               app.kubernetes.io/name=sessiondb
               role=primary
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      8Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       sessiondb-0
Events:        <none>

---

% kubectl describe pvc data-sessiondb-0
Name:          data-sessiondb-0
Namespace:     default
StorageClass:  standard
Status:        Bound
Volume:        pvc-6b56b20c-3e56-4a92-9278-794bf6cda4de
Labels:        app.kubernetes.io/instance=asdf
               app.kubernetes.io/name=sessiondb
               role=primary
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      8Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       sessiondb-0
Events:        <none>
cyrill@CyrillsMBP4380 core % kubectl describe pv pvc-6b56b20c-3e56-4a92-9278-794bf6cda4de
Name:            pvc-6b56b20c-3e56-4a92-9278-794bf6cda4de
Labels:          <none>
Annotations:     hostPathProvisionerIdentity: 10bfa079-1086-4a77-849e-7d00de8e34dc
                 pv.kubernetes.io/provisioned-by: k8s.io/minikube-hostpath
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    standard
Status:          Bound
Claim:           default/data-sessiondb-0
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        8Gi
Node Affinity:   <none>
Message:         
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /tmp/hostpath-provisioner/default/data-sessiondb-0
    HostPathType:  
Events:            <none>

编辑:添加 DEBUG 级别的日志 --> 希望修复:directory "/bitnami/postgresql/data" is not a database cluster directory

EDIT2:添加初始化容器的日志volumePermissions.enabled和权限

EDIT3:好的,所以我创建了一个新版本来比较新旧版本。我想知道,为什么文件中存在如此大的差异,因为两个文件都在工作,直到一个文件不再工作。 (正常营业期间,没有升级,什么都没有。)

我真的希望没有其他人 运行 解决这个问题,但最后我发现了问题,这一次不仅是在椅子和显示器之间,而且还涉及 RTFM。

如前所述,我正在使用 minikube 运行 我的 k8s 集群,它提供存储在主机磁盘上的 PV。你可能会问它存储在哪里?非常准确,这里:/tmp/hostpath-provisioner/default/data-sessiondb-0/data/。你发现问题了吗?不,我也花了一些时间来弄清楚。为什么 minikube 在地球上使用 tmp 文件夹来存储 persistant volume claims?

此文件夹自动生成。不时清除。

解决方案:更改路径并且不要将 PV 存储在 tmp 个文件夹。

他们在这里提到了这一点:https://minikube.sigs.k8s.io/docs/handbook/persistent_volumes/#a-note-on-mounts-persistence-and-minikube-hosts 并举了一个例子。

但是为什么默认使用“危险的”tmp 路径而不是 data 而不在其中放置 警告 横幅?

唉。结束这个问题^^

--> 解决方法:https://github.com/kubernetes/minikube/issues/7511#issuecomment-612099413


Github 个问题:

我的 Github 问题需要在文档中澄清:https://github.com/kubernetes/minikube/issues/13038#issuecomment-981821696