Terraform Bigquery 创建 tables 替换 table 而不是编辑

Question

我添加了一个 json 文件，其中包含我要创建的所有 tables :

tables.json


    "tables": {
            "table1": {
                "dataset_id": "dataset1",
                "table_id": "table1",
                "schema_path": "folder/table1.json"
            },
            "table2": {
                "dataset_id": "dataset2",
                "table_id": "table2",
                "schema_path": "folder/table2.json"
            }
    }

然后在 Terraform 资源上使用 foreach，我想动态创建这些 tables :

local.tf 文件


    locals {
      tables = jsondecode(file("${path.module}/resource/tables.json"))["tables"]
    }

variables.tf 文件


    variable "project_id" {
      description = "Project ID, used to enforce providing a project id."
      type = string
    }
    
    variable "time_partitioning" {
      description = "Configures time-based partitioning for this table."
      type = map(string)
      default = {
        type = "DAY"
        field = "my_field"
      }
    }

main.tf 文件


    resource "google_bigquery_table" "tables" {
      for_each = local.tables
      project = var.project_id
      dataset_id = each.value["dataset_id"]
      table_id = each.value["table_id"]
    
      dynamic "time_partitioning" {
        for_each = [
          var.time_partitioning
        ]
        content {
          type = try(each.value["partition_type"], time_partitioning.value["type"])
          field = try(each.value["partition_field"], time_partitioning.value["field"])
          expiration_ms = try(time_partitioning.value["expiration_ms"], null)
          require_partition_filter = try(time_partitioning.value["require_partition_filter"], null)
        }
      }
    
      schema = file("${path.module}/resource/schema/${each.value["schema_path"]}")
    }

架构文件包含经典的 bigquery 架构，例如：


    [
        {
            "name": "field",
            "type": "STRING",
            "mode": "NULLABLE",
            "description": "My field"
        }
    ]

tables 的创建效果很好，但是当我在模式上添加一个新的可为 null 的字段时，Terraform 建议“替换 table”（销毁并重新创建）而不是“更新 table".

本机 Bigquery 和 Terraform 在这种情况下的正常行为是更新 table。

当使用相同的 Terraform 资源但没有 forEach 进行相同的测试时，Terraform 具有预期的行为并建议更新 table。

带有“forEach”的 Terraform 日志示例：

 

    # google_bigquery_table.tables["table1"] must be replaced
    -/+ resource "google_bigquery_table" "tables" {
          ~ creation_time       = 1616764894477 -> (known after apply)
            dataset_id          = "dataset1"
            deletion_protection = true
          ~ etag                = "G9qwId8jgQS8nN4N61zqcA==" -> (known after apply)
          ~ expiration_time     = 0 -> (known after apply)
          ~ id                  = "projects/my-project/datasets/dataset1/tables/table1" -> (known after apply)
          - labels              = {} -> null
          ~ last_modified_time  = 1617075251337 -> (known after apply)
          ~ location            = "EU" -> (known after apply)
          ~ num_bytes           = 0 -> (known after apply)
          ~ num_long_term_bytes = 0 -> (known after apply)
          ~ num_rows            = 0 -> (known after apply)
            project             = "project"
          ~ schema              = jsonencode(
              ~ [ # forces replacement
                    {
                        description = "Field"
                        mode        = "NULLABLE"
                        name        = "field"
                        type        = "STRING"
                    }
             
             .....
             + {
                  + description = "Field"
                  + mode        = "NULLABLE"
                  + name        = "newField"
                  + type        = "STRING"
               }

Terraform 正确显示并检测要为 table 添加的新列，但指示替换而不是版本。

我再说一遍，在没有 forEach 的情况下使用相同的 Terraform 资源并在单个 Bigquery table 上进行的完全相同的测试效果很好（相同的模式，相同的更改）。我创建了 table 并在添加新的可为空列时，Terraform 提出了一个版本（预期行为）。

我查看了 Terraform 文档和网络，没有看到一些使用 Terraform 管理 table 列表的示例。

是否无法使用已配置的 table 和 foreach 创建和更新 table？

感谢您的帮助。

Answer 1

这听起来像是提供商错误。我发现这个 issue in the terraform-provider-google repository that seems related to your issue. The fix 是在 13 小时前（在撰写本文时）合并的。所以，也许您可以等待下一个版本 (v3.63.0)，看看它是否能解决您的问题。

仅供参考：您可能想要验证修复提交是否确实包含在下一个版本中。之前遇到过在release之前合并到master里面的东西，实际上并没有发布

Answer 2

非常感谢@Alessandro，问题确实是由于 Terraform 提供的 Google 版本。我用的是v3.62.0版本的Google provider，你给我指个好方向

我也看到了这个link：https://github.com/hashicorp/terraform-provider-google/issues/8503

“tpolekhin”（感谢他）发表了非常有用的评论：

Hopefully im not beating a dead horse commenting on the closed issue, but I did some testing with various versions on the provider, and it behaves VERY differently each time.

So, our terraform code change was pretty simple: add 2 new columns to existing BigQuery table SCHEDULE
Nothing changes between runs - only provider version

v3.52.0
Plan: 0 to add, 19 to change, 0 to destroy.
Mostly adds + mode = "NULLABLE" to fields in bunch of tables, and adds 2 new fields in SCHEDULE table

v3.53.0
Plan: 0 to add, 2 to change, 0 to destroy.
Adds 2 new fields to SCHEDULE table, and moves one field in another table in a different place (sorting?)

v3.54.0
Plan: 1 to add, 1 to change, 1 to destroy.
Adds 2 new fields to SCHEDULE table, and moves one field in another table in a different place (sorting?) but now with table re-creation for some reason

v3.55.0
Plan: 0 to add, 2 to change, 0 to destroy.
Adds 2 new fields to SCHEDULE table, and moves one field in another table in a different place (sorting?)
behaves exactly like v3.53.0

v3.56.0
Plan: 1 to add, 0 to change, 1 to destroy.

在此评论中，我们可以看到某些版本存在问题。例如，这适用于 v3.55.0 但不适用于 v3.56.0

我暂时将版本降到v3.55.0，等下个版本解决这个问题，我再升级。

provider.tf :


    provider "google" {
      version = "= 3.55.0"
    }

Terraform Bigquery 创建 tables 替换 table 而不是编辑

Terraform Bigquery create tables replace table instead of edit

google-bigquery

google-cloud-platform

terraform

terraform-provider-gcp