Nomad constraint "${attr.vault.version} version >= 0.6.1" 访问保险库

Question

我正在尝试部署一个 Nomad 作业，它有一个模板，可以从 Vault 中获取一些秘密。

我的问题是，由于约束，它一直显示 Placement Failure，我不明白为什么：

Constraint ${attr.vault.version} version >= 0.6.1 filtered 1 node

游牧配置

datacenter = "dc1"
data_dir = "/var/lib/nomad"
advertise {
  # Defaults to the first private IP address.
  http = "10.134.43.195"
  rpc  = "10.134.43.195"
  serf = "10.134.43.195"
}

server {
  enabled = true
  bootstrap_expect = 1
  server_join {
    retry_join = ["provider=digitalocean api_token=[SECRET] tag_name=nomad_auto_join"],
  }
}
client {
  enabled = true
}
# Consul is installed locally and clustered
consul {
  address = "http://127.0.0.1:8500"
  server_auto_join = true
  client_auto_join = true
  auto_advertise = true
}
vault {
  enabled = true
  address = "http://vault.service.consul:8200"
  token = "[NOMAD_VAULT_TOKEN]"
  create_from_role = "nomad-cluster"
  task_token_ttl = "1h"
}
autopilot {
    cleanup_dead_servers = true
    last_contact_threshold = "200ms"
    max_trailing_logs = 250
    server_stabilization_time = "10s"
    enable_redundancy_zones = false
    disable_upgrade_migration = false
    enable_custom_upgrades = false
}
telemetry {
  publish_allocation_metrics = true
  publish_node_metrics = true
  prometheus_metrics = true
}

### Nomad Token to access Vault

NOMAD_VAULT_TOKEN 使用此命令生成：

vault token create -policy nomad-server -period 72h -orphan

保管库策略游牧服务器

nomad-server 保险库政策如下：

# Allow creating tokens under "nomad-cluster" token role.Z
path "auth/token/create/nomad-cluster" {
  capabilities = ["update"]
}

# Allow looking up "nomad-cluster" token role.
path "auth/token/roles/nomad-cluster" {
  capabilities = ["read"]
}

# Allow looking up the token passed to Nomad to validate # the token has the
# proper capabilities. This is provided by the "default" policy.
path "auth/token/lookup-self" {
  capabilities = ["read"]
}

# Allow looking up incoming tokens to validate they have permissions to access
# the tokens they are requesting. This is only required if
# `allow_unauthenticated` is set to false.
path "auth/token/lookup" {
  capabilities = ["update"]
}

# Allow revoking tokens that should no longer exist. This allows revoking
# tokens for dead tasks.
path "auth/token/revoke-accessor" {
  capabilities = ["update"]
}

# Allow checking the capabilities of our own token. This is used to validate the
# token upon startup.
path "sys/capabilities-self" {
  capabilities = ["update"]
}

# Allow our own token to be renewed.
path "auth/token/renew-self" {
  capabilities = ["update"]
}

# This is where needed secretes are fetched from
path "kv/*" {
  capabilities = ["update", "read", "create"]
}

游牧工作定义

我的 Nomad 工作定义是：

job "api" {
  datacenters = ["dc1"]
  type = "service"



  group "api" {
    count = 1
    update {
      max_parallel = 1
      min_healthy_time = "30s"
      healthy_deadline = "10m"
      progress_deadline = "11m"
      auto_revert = true
    }
    task "api" {
      driver = "docker"
      config {
        image = "registry.gitlab.com/[GROUP]/[PROJECT]/${ENVIRONMENT}:${BUILD_NUMBER}"
        port_map {
          nginx = 80
        }
        auth {
          username = "${REGISTRY_USER}"
          password = "${REGISTRY_PASS}"
        }
        force_pull = true
        hostname = "api"
      }

      vault {
        policies = ["kv"]

        change_mode   = "signal"
        change_signal = "SIGINT"
        //    env = "false"
      }

      template {
        data        = <<EOT
APP_NAME={{ key "services/api/app/${ENVIRONMENT}/APP_NAME" }}
APP_ENV={{ key "services/api/app/${ENVIRONMENT}/APP_ENV" }}
APP_KEY={{with secret "kv/services/api/app/${ENVIRONMENT}"}}{{.Data.APP_KEY.value}}{{end}}
APP_DEBUG={{ key "services/api/app/${ENVIRONMENT}/APP_DEBUG" }}
APP_URL={{ key "services/api/app/${ENVIRONMENT}/APP_URL" }}

LOG_CHANNEL={{ key "services/api/log/${ENVIRONMENT}/LOG_CHANNEL" }}

DB_CONNECTION={{ key "services/api/db/${ENVIRONMENT}/DB_CONNECTION" }}
DB_HOST={{ key "services/api/db/${ENVIRONMENT}/DB_HOST" }}
DB_PORT={{ key "services/api/db/${ENVIRONMENT}/DB_PORT" }}
DB_DATABASE={{with secret "kv/services/api/db/${ENVIRONMENT}/DB_DATABASE"}}{{.Data.value}}{{end}}
DB_USERNAME={{with secret "kv/services/api/db/${ENVIRONMENT}/DB_USERNAME"}}{{.Data.value}}{{end}}
DB_PASSWORD={{with secret "kv/services/api/db/${ENVIRONMENT}/DB_PASSWORD"}}{{.Data.value}}{{end}}

BROADCAST_DRIVER={{ key "services/api/broadcast/${ENVIRONMENT}/BROADCAST_DRIVER" }}
CACHE_DRIVER={{ key "services/api/cache/${ENVIRONMENT}/CACHE_DRIVER" }}
SESSION_DRIVER={{ key "services/api/session/${ENVIRONMENT}/SESSION_DRIVER" }}
SESSION_LIFETIME={{ key "services/api/session/${ENVIRONMENT}/SESSION_LIFETIME" }}
QUEUE_DRIVER={{ key "services/api/queue/${ENVIRONMENT}/QUEUE_DRIVER" }}

REDIS_HOST={{ key "services/api/redis/${ENVIRONMENT}/REDIS_HOST" }}
REDIS_PORT={{ key "services/api/redis/${ENVIRONMENT}/REDIS_PORT" }}
REDIS_PASSWORD={{with secret "kv/services/api/redis/${ENVIRONMENT}/REDIS_PASSWORD"}}{{.Data.value}}{{end}}

MAIL_DRIVER={{ key "services/api/mail/${ENVIRONMENT}/MAIL_DRIVER" }}
MAIL_HOST={{ key "services/api/mail/${ENVIRONMENT}/MAIL_HOST" }}
MAIL_PORT={{ key "services/api/mail/${ENVIRONMENT}/MAIL_PORT" }}
MAIL_USERNAME={{ key "services/api/mail/${ENVIRONMENT}/MAIL_USERNAME" }}
MAIL_PASSWORD={{ key "services/api/mail/${ENVIRONMENT}/MAIL_PASSWORD" }}
MAIL_ENCRYPTION={{ key "services/api/mail/${ENVIRONMENT}/MAIL_ENCRYPTION" }}
MAIL_FROM_ADDRESS={{ key "services/api/mail/${ENVIRONMENT}/MAIL_FROM_ADDRESS" }}
MAIL_FROM_NAME={{ key "services/api/mail/${ENVIRONMENT}/MAIL_FROM_NAME" }}

MAILGUN_DOMAIN={{ key "services/api/mailgun/${ENVIRONMENT}/MAILGUN_DOMAIN" }}
MAILGUN_SECRET={{with secret "kv/services/api/mailgun/${ENVIRONMENT}/MAILGUN_SECRET"}}{{.Data.value}}{{end}}

DO_SPACES_KEY={{with secret "kv/services/api/spaces/${ENVIRONMENT}/DO_SPACES_KEY"}}{{.Data.value}}{{end}}
DO_SPACES_SECRET={{with secret "kv/services/api/spaces/${ENVIRONMENT}/DO_SPACES_SECRET"}}{{.Data.value}}{{end}}
DO_SPACES_ENDPOINT={{ key "services/api/spaces/${ENVIRONMENT}/DO_SPACES_ENDPOINT" }}
DO_SPACES_REGION={{ key "services/api/spaces/${ENVIRONMENT}/DO_SPACES_REGION" }}
DO_SPACES_BUCKET={{ key "services/api/spaces/${ENVIRONMENT}/DO_SPACES_BUCKET" }}

JWT_SECRET={{with secret "kv/services/api/jwt/${ENVIRONMENT}"}}{{.Data.JWT_SECRET}}{{end}}
EOT
        destination   = "custom/.env"
//        change_mode   = "signal"
//        change_signal = "SIGINT"
        env         = true
      }

      service {
        name = "api"
        tags = [
          "urlprefix-${ENVIRONMENT_URL}/"
        ]
        port = "nginx"
        check {
          type = "tcp"
          port = "nginx"
          interval = "10s"
          timeout = "2s"
        }
      }
      resources {
        cpu    = 500 
        memory = 256
        network {
          mbits = 100
          port "nginx" {}
        }
      }
    }
  }
}

Nomad 日志

在 Nomad 日志中，我可以检查它是否正确地从 Vault 获取令牌：

Oct 16 13:29:19 nomad-server-0 nomad: 2019-10-16T13:29:19.333Z [INFO ] client.fingerprint_mgr.vault: Vault is available
Oct 16 13:29:19 nomad-server-0 nomad: 2019-10-16T13:29:19.334Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=vault period=15s
Oct 16 13:29:19 nomad-server-0 nomad: 2019-10-16T13:29:19.354Z [DEBUG] nomad.vault: starting renewal loop: creation_ttl=72h0m0s
Oct 16 13:29:19 nomad-server-0 nomad: 2019-10-16T13:29:19.375Z [DEBUG] client.fingerprint_mgr: detected fingerprints: node_attrs="[arch cgroup consul cpu host network nomad signal storage vault]"
Oct 16 13:29:19 nomad-server-0 nomad: 2019-10-16T13:29:19.380Z [DEBUG] nomad.vault: successfully renewed server token
Oct 16 13:29:19 nomad-server-0 nomad: 2019-10-16T13:29:19.380Z [INFO ] nomad.vault: successfully renewed token: next_renewal=35h59m59.999975432s

我卡在这里了，如果有人能提供任何见解，我将不胜感激！

[编辑] 我正在使用 Vault v1.2.0-rc1.

Answer 1

根据 docs、Note: Vault integration requires Vault version 0.6.2 or higher.

您的错误消息支持这一点 -- Constraint ${attr.vault.version} version >= 0.6.1 filtered 1 node。通常，约束是您在游牧规范中指定的内容，但在这种情况下，它看起来像是来自游牧规范。

我想你需要 upgrade vault 到 0.6.2

Answer 2

我已将 Vault 升级到版本 v1.2.3，它开始工作了。

Answer 3

有时这个错误意味着 vault 仍然是密封的或者在 vault 重启后已经密封

这可以通过在保险库 dns 地址上执行 dig 来测试，例如 vault.service.consul

Nomad constraint "${attr.vault.version} version >= 0.6.1" 访问保险库

Nomad constraint "${attr.vault.version} version >= 0.6.1" to access vault

integration

token

hashicorp-vault

nomad

游牧配置

保管库策略游牧服务器

游牧工作定义

Nomad 日志