操作员获取部署配置的最佳实践

Question

我在做operator-sdk，在controller中，我们经常需要创建一个Deployment对象，而Deployment资源有很多配置项，比如环境变量或者ports定义等等。我想知道获取这些值的最佳方法是什么，我不想对它们进行硬编码，例如 variable_a 或 variable_b.

或许，你可以将它们作为规范放入CRD中，然后将它们传递给Operator Controller；或者你可以将它们放在 configmap 中，然后将 configmap 名称传递给 Operator Controller，Operator Controller 可以访问 configmap 来获取它们；或者你可以放入模板文件，然后在操作员控制器中，控制器必须读取该模板文件。

处理这种情况的最佳方法或最佳做法是什么？感谢分享您的想法或观点。

    deployment := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      m.Name,
            Namespace: m.Namespace,
            Labels:    ls,
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: ls,
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: ls,
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{{
                        Image: "....",
                        Name: m.Name,
                        Ports: []corev1.ContainerPort{{
                            ContainerPort: port_a,
                            Name:          "tcpport",
                        }},
                        Env: []corev1.EnvVar{
                            {
                                Name:  "aaaa",
                                Value: variable_a,
                            },
                            {
                                Name:  "bbbb",
                                Value: variable_b,
                            },

Answer 1

使用环境变量

您的应用可以方便地将您的数据作为环境变量获取。

来自 `ConfigMap`

的环境变量

对于 non-sensitive 数据，您可以将变量存储在 ConfigMap 中，然后使用 ConfigMap 数据定义容器环境变量。

Example from Kubernetes docs:

首先创建 ConfigMap。文件 configmaps.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: special-config
  namespace: default
data:
  special.how: very
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: env-config
  namespace: default
data:
  log_level: INFO

创建 ConfigMap：

kubectl create -f ./configmaps.yaml

然后在Pod规范中定义环境变量，pod-multiple-configmap-env-variable.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: dapi-test-pod
spec:
  containers:
    - name: test-container
      image: k8s.gcr.io/busybox
      command: [ "/bin/sh", "-c", "env" ]
      env:
        - name: SPECIAL_LEVEL_KEY
          valueFrom:
            configMapKeyRef:
              name: special-config
              key: special.how
        - name: LOG_LEVEL
          valueFrom:
            configMapKeyRef:
              name: env-config
              key: log_level
  restartPolicy: Never

创建 Pod:

kubectl create -f ./pod-multiple-configmap-env-variable.yaml

现在，在您的控制器中，您可以读取这些环境变量 SPECIAL_LEVEL_KEY（它将为您提供来自 special-config ConfigMap 的 special.how 值）和 LOG_LEVEL (这会给你 log_level 来自 env-config ConfigMap 的价值：

例如：

specialLevelKey := os.Getenv("SPECIAL_LEVEL_KEY")
logLevel := os.Getenv("LOG_LEVEL")

fmt.Println("SPECIAL_LEVEL_KEY:", specialLevelKey)
fmt.Println("LOG_LEVEL:", logLevel)

来自 `Secret`

的环境变量

如果您的数据是敏感的，您可以将其存储在 Secret 中，然后使用 Secret 作为环境变量。

To create a Secret manually:

您首先需要使用 base64.

对您的字符串进行编码

# encode username
$ echo -n 'admin' | base64
YWRtaW4=

# encode password
$ echo -n '1f2d1e2e67df' | base64
MWYyZDFlMmU2N2Rm

然后用上面的数据创建一个Secret：

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm

用 kubectl apply 创建一个 Secret:

$ kubectl apply -f ./secret.yaml

请注意，还有其他创建密文的方法，请选择最适合您的方法：

现在您可以use this created Secret for environment variables。

To use a secret in an environment variable in a Pod:

Create a secret or use an existing one. Multiple Pods can reference the same secret.

Modify your Pod definition in each container that you wish to consume the value of a secret key to add an environment variable for each secret key you wish to consume. The environment variable that consumes the secret key should populate the secret's name and key in env[].valueFrom.secretKeyRef.

Modify your image and/or command line so that the program looks for values in the specified environment variables.

这是来自 Kubernetes 文档的 Pod 示例，展示了如何使用 Secret 环境变量：

apiVersion: v1
kind: Pod
metadata:
  name: secret-env-pod
spec:
  containers:
  - name: mycontainer
    image: redis
    env:
      - name: SECRET_USERNAME
        valueFrom:
          secretKeyRef:
            name: mysecret
            key: username
      - name: SECRET_PASSWORD
        valueFrom:
          secretKeyRef:
            name: mysecret
            key: password
  restartPolicy: Never

最后，如文档中所述：

Inside a container that consumes a secret in an environment variables, the secret keys appear as normal environment variables containing the base64 decoded values of the secret data.

现在，在您的控制器中，您可以读取这些环境变量 SECRET_USERNAME（它将为您提供来自 mysecret Secret 的 username 值）和 SECRET_PASSWORD (这将为您提供来自 mysecret Secret 的 password 值：

例如：

username := os.Getenv("SECRET_USERNAME")
password := os.Getenv("SECRET_PASSWORD")

使用卷

您还可以将 ConfigMap 和 Secret 作为一个卷挂载给您 pods。

Populate a Volume with data stored in a ConfigMap:

apiVersion: v1
kind: Pod
metadata:
  name: dapi-test-pod
spec:
  containers:
    - name: test-container
      image: k8s.gcr.io/busybox
      command: [ "/bin/sh", "-c", "ls /etc/config/" ]
      volumeMounts:
      - name: config-volume
        mountPath: /etc/config
  volumes:
    - name: config-volume
      configMap:
        # Provide the name of the ConfigMap containing the files you want
        # to add to the container
        name: special-config
  restartPolicy: Never

Using Secrets as files from a Pod:

To consume a Secret in a volume in a Pod:

Create a secret or use an existing one. Multiple Pods can reference the same secret.

Modify your Pod definition to add a volume under .spec.volumes[]. Name the volume anything, and have a .spec.volumes[].secret.secretName field equal to the name of the Secret object.

Add a .spec.containers[].volumeMounts[] to each container that needs the secret. Specify .spec.containers[].volumeMounts[].readOnly = true and .spec.containers[].volumeMounts[].mountPath to an unused directory name where you would like the secrets to appear. Modify your image or command line so that the program looks for files in that directory. Each key in the secret data map becomes the filename under mountPath.

在卷中安装 Secret 的 Pod 示例：

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: redis
    volumeMounts:
    - name: foo
      mountPath: "/etc/foo"
      readOnly: true
  volumes:
  - name: foo
    secret:
      secretName: mysecret

操作员获取部署配置的最佳实践

Best Practice for Operators for how to get Deployment's configuration

kubernetes

kubernetes-custom-resources

operator-sdk

使用环境变量

来自 `ConfigMap`

来自 `Secret`

使用卷

操作员获取部署配置的最佳实践

Best Practice for Operators for how to get Deployment's configuration

kubernetes

kubernetes-custom-resources

operator-sdk

使用环境变量

来自 ConfigMap

来自 Secret

使用卷

来自 `ConfigMap`

来自 `Secret`