从 CRATE:4.2.7 升级到 CRATE:4.3.4 时,CRATE 数据库表导致错误

CRATE db tables are causing an error when upgrading to CRATE:4.3.4 from CRATE:4.2.7

恢复快照或将 cratedb 从 4.2.7 升级到更高版本 4.3.4 时遇到以下错误

SnapshotRestoreException[[repo_name:snapshot_name/******] cannot restore index [schema_name.table_name] because it cannot be upgraded]
org.elasticsearch.snapshots.SnapshotRestoreException: [repo_name:snapshot_name/******] cannot restore index [schema_name.table_name] because it cannot be upgraded
    at org.elasticsearch.snapshots.RestoreService.execute(RestoreService.java:273)
    at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)
    at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:681)
    at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:307)
    at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:207)
    at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:139)
    at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:153)
    at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:191)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:258)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:221)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
    at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.IllegalStateException: unable to upgrade the mappings for the index [[repo_name:snapshot_name/******]]
    at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkMappingsCompatibility(MetadataIndexUpgradeService.java:175)
    at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:98)
    at org.elasticsearch.snapshots.RestoreService.execute(RestoreService.java:270)
    ... 12 more
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed to parse mapping [default]: Mapping definition for [dynamicFieldName3] has unsupported parameters:  [fields : {keyword={type=keyword, ignore_above=256}}]
    at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:285)
    at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:272)
    at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:251)
    at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkMappingsCompatibility(MetadataIndexUpgradeService.java:171)
    ... 14 more
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Mapping definition for [dynamicFieldName3] has unsupported parameters:  [fields : {keyword={type=keyword, ignore_above=256}}]
    at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:153)
    at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:147)
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:215)
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:163)
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:134)
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:207)
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:163)
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:134)
    at org.elasticsearch.index.mapper.ArrayTypeParser.parse(ArrayTypeParser.java:51)
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:207)
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:163)
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:134)
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:207)
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:163)
    at org.elasticsearch.index.mapper.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:103)
    at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:101)
    at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:89)
    at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:283)

下面是示例 table 创建架构。

CREATE TABLE IF NOT EXISTS TABLE_NAME (
  "id" STRING,
  "field1" STRING NOT NULL,
  "field2" STRING NOT NULL,
  "field3" OBJECT NOT NULL,
  "createdAt" TIMESTAMP NOT NULL,
  "updatedAt" TIMESTAMP NOT NULL,
  PRIMARY KEY ("field1", "field2")
)

由于 field3 是动态对象,它允许向其中添加更多属性。

所以,现在将数据插入table后,当前架构如下

CREATE TABLE IF NOT EXISTS TABLE_NAME (
  "id" STRING,
  "field1" STRING NOT NULL,
  "field2" STRING NOT NULL,
  "field3" OBJECT(DYNAMIC) AS (
      "dynamicField1" BOOLEAN,
      "dynamicField2" TEXT,
      "dynamicField3" TEXT INDEX USING FULLTEXT,
      "dynamicField4" TEXT
   )  NOT NULL,
  "createdAt" TIMESTAMP NOT NULL,
  "updatedAt" TIMESTAMP NOT NULL,
  PRIMARY KEY ("field1", "field2")
)
...
...
...

我拍了一张快照,现在我正在尝试 restore/upgrade 到更高版本,它面临上述错误。

更新 1:

SELECT DISTINCT version['created'], version['upgraded'] FROM "information_schema"."tables" WHERE table_schema not IN ('sys','information_schema','pg_catalog') limit 100;

Result from query:
version['created']  version['upgraded']
4.0.12              4.2.7 
4.1.5               4.2.7 
4.2.7               NULL 
SELECT distinct min_lucene_version FROM sys.shards limit 100;

Result from query:

min_lucene_version
8.0.0 
8.5.1 
8.4.0

上述错误

[fields : {keyword={type=keyword, ignore_above=256}}]

提示,您想要 upgrade/restore 使用 CrateDB <3.0 版本创建的 table / 快照到 CrateDB 4.x

CrateDB 4.x 仅支持使用 3.x 及以上创建的 tables CrateDB 3.x 仅支持使用 2.x 及以上

创建的 table

快照基本上只是 Lucene 的一个副本 indexes/segments

如评论中所述,您应该 运行 以下查询来检查 table 是使用哪个版本创建的:

SELECT DISTINCT version['created'], version['upgraded'] FROM "information_schema"."tables" WHERE table_schema not IN ('sys','information_schema','pg_catalog')
SELECT distinct min_lucene_version FROM sys.shards

如果只有较旧的 table 受到影响,我建议使用 4.2.7 重新索引那些 table,然后 运行 升级

SHOW CREATE table old;
INSERT INTO new SELECT * FROM old;
--or
COPY old TO;
COPY new FROM;