从 CRATE:4.2.7 升级到 CRATE:4.3.4 时，CRATE 数据库表导致错误
CRATE db tables are causing an error when upgrading to CRATE:4.3.4 from CRATE:4.2.7
恢复快照或将 cratedb 从 4.2.7 升级到更高版本 4.3.4 时遇到以下错误
SnapshotRestoreException[[repo_name:snapshot_name/******] cannot restore index [schema_name.table_name] because it cannot be upgraded]
org.elasticsearch.snapshots.SnapshotRestoreException: [repo_name:snapshot_name/******] cannot restore index [schema_name.table_name] because it cannot be upgraded
at org.elasticsearch.snapshots.RestoreService.execute(RestoreService.java:273)
at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:681)
at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:307)
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:207)
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:139)
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:153)
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:191)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:258)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:221)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.IllegalStateException: unable to upgrade the mappings for the index [[repo_name:snapshot_name/******]]
at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkMappingsCompatibility(MetadataIndexUpgradeService.java:175)
at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:98)
at org.elasticsearch.snapshots.RestoreService.execute(RestoreService.java:270)
... 12 more
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed to parse mapping [default]: Mapping definition for [dynamicFieldName3] has unsupported parameters: [fields : {keyword={type=keyword, ignore_above=256}}]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:285)
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:272)
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:251)
at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkMappingsCompatibility(MetadataIndexUpgradeService.java:171)
... 14 more
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Mapping definition for [dynamicFieldName3] has unsupported parameters: [fields : {keyword={type=keyword, ignore_above=256}}]
at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:153)
at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:147)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:215)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:163)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:134)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:207)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:163)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:134)
at org.elasticsearch.index.mapper.ArrayTypeParser.parse(ArrayTypeParser.java:51)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:207)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:163)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:134)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:207)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:163)
at org.elasticsearch.index.mapper.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:103)
at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:101)
at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:89)
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:283)
下面是示例 table 创建架构。
CREATE TABLE IF NOT EXISTS TABLE_NAME (
"id" STRING,
"field1" STRING NOT NULL,
"field2" STRING NOT NULL,
"field3" OBJECT NOT NULL,
"createdAt" TIMESTAMP NOT NULL,
"updatedAt" TIMESTAMP NOT NULL,
PRIMARY KEY ("field1", "field2")
)
由于 field3 是动态对象，它允许向其中添加更多属性。
所以，现在将数据插入table后，当前架构如下
CREATE TABLE IF NOT EXISTS TABLE_NAME (
"id" STRING,
"field1" STRING NOT NULL,
"field2" STRING NOT NULL,
"field3" OBJECT(DYNAMIC) AS (
"dynamicField1" BOOLEAN,
"dynamicField2" TEXT,
"dynamicField3" TEXT INDEX USING FULLTEXT,
"dynamicField4" TEXT
) NOT NULL,
"createdAt" TIMESTAMP NOT NULL,
"updatedAt" TIMESTAMP NOT NULL,
PRIMARY KEY ("field1", "field2")
)
...
...
...
我拍了一张快照，现在我正在尝试 restore/upgrade 到更高版本，它面临上述错误。
更新 1：
SELECT DISTINCT version['created'], version['upgraded'] FROM "information_schema"."tables" WHERE table_schema not IN ('sys','information_schema','pg_catalog') limit 100;
Result from query:
version['created'] version['upgraded']
4.0.12 4.2.7
4.1.5 4.2.7
4.2.7 NULL
SELECT distinct min_lucene_version FROM sys.shards limit 100;
Result from query:
min_lucene_version
8.0.0
8.5.1
8.4.0
上述错误
[fields : {keyword={type=keyword, ignore_above=256}}]
提示，您想要 upgrade/restore 使用 CrateDB <3.0 版本创建的 table / 快照到 CrateDB 4.x
CrateDB 4.x 仅支持使用 3.x 及以上创建的 tables
CrateDB 3.x 仅支持使用 2.x 及以上
创建的 table
快照基本上只是 Lucene 的一个副本 indexes/segments
如评论中所述，您应该 运行 以下查询来检查 table 是使用哪个版本创建的：
SELECT DISTINCT version['created'], version['upgraded'] FROM "information_schema"."tables" WHERE table_schema not IN ('sys','information_schema','pg_catalog')
SELECT distinct min_lucene_version FROM sys.shards
如果只有较旧的 table 受到影响，我建议使用 4.2.7 重新索引那些 table，然后 运行 升级
SHOW CREATE table old;
INSERT INTO new SELECT * FROM old;
--or
COPY old TO;
COPY new FROM;
恢复快照或将 cratedb 从 4.2.7 升级到更高版本 4.3.4 时遇到以下错误
SnapshotRestoreException[[repo_name:snapshot_name/******] cannot restore index [schema_name.table_name] because it cannot be upgraded]
org.elasticsearch.snapshots.SnapshotRestoreException: [repo_name:snapshot_name/******] cannot restore index [schema_name.table_name] because it cannot be upgraded
at org.elasticsearch.snapshots.RestoreService.execute(RestoreService.java:273)
at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:681)
at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:307)
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:207)
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:139)
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:153)
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:191)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:258)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:221)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.IllegalStateException: unable to upgrade the mappings for the index [[repo_name:snapshot_name/******]]
at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkMappingsCompatibility(MetadataIndexUpgradeService.java:175)
at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:98)
at org.elasticsearch.snapshots.RestoreService.execute(RestoreService.java:270)
... 12 more
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed to parse mapping [default]: Mapping definition for [dynamicFieldName3] has unsupported parameters: [fields : {keyword={type=keyword, ignore_above=256}}]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:285)
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:272)
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:251)
at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkMappingsCompatibility(MetadataIndexUpgradeService.java:171)
... 14 more
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Mapping definition for [dynamicFieldName3] has unsupported parameters: [fields : {keyword={type=keyword, ignore_above=256}}]
at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:153)
at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:147)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:215)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:163)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:134)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:207)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:163)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:134)
at org.elasticsearch.index.mapper.ArrayTypeParser.parse(ArrayTypeParser.java:51)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:207)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:163)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:134)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:207)
at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:163)
at org.elasticsearch.index.mapper.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:103)
at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:101)
at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:89)
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:283)
下面是示例 table 创建架构。
CREATE TABLE IF NOT EXISTS TABLE_NAME (
"id" STRING,
"field1" STRING NOT NULL,
"field2" STRING NOT NULL,
"field3" OBJECT NOT NULL,
"createdAt" TIMESTAMP NOT NULL,
"updatedAt" TIMESTAMP NOT NULL,
PRIMARY KEY ("field1", "field2")
)
由于 field3 是动态对象，它允许向其中添加更多属性。
所以，现在将数据插入table后，当前架构如下
CREATE TABLE IF NOT EXISTS TABLE_NAME (
"id" STRING,
"field1" STRING NOT NULL,
"field2" STRING NOT NULL,
"field3" OBJECT(DYNAMIC) AS (
"dynamicField1" BOOLEAN,
"dynamicField2" TEXT,
"dynamicField3" TEXT INDEX USING FULLTEXT,
"dynamicField4" TEXT
) NOT NULL,
"createdAt" TIMESTAMP NOT NULL,
"updatedAt" TIMESTAMP NOT NULL,
PRIMARY KEY ("field1", "field2")
)
...
...
...
我拍了一张快照，现在我正在尝试 restore/upgrade 到更高版本，它面临上述错误。
更新 1：
SELECT DISTINCT version['created'], version['upgraded'] FROM "information_schema"."tables" WHERE table_schema not IN ('sys','information_schema','pg_catalog') limit 100;
Result from query:
version['created'] version['upgraded']
4.0.12 4.2.7
4.1.5 4.2.7
4.2.7 NULL
SELECT distinct min_lucene_version FROM sys.shards limit 100;
Result from query:
min_lucene_version
8.0.0
8.5.1
8.4.0
上述错误
[fields : {keyword={type=keyword, ignore_above=256}}]
提示，您想要 upgrade/restore 使用 CrateDB <3.0 版本创建的 table / 快照到 CrateDB 4.x
CrateDB 4.x 仅支持使用 3.x 及以上创建的 tables CrateDB 3.x 仅支持使用 2.x 及以上
创建的 table快照基本上只是 Lucene 的一个副本 indexes/segments
如评论中所述，您应该 运行 以下查询来检查 table 是使用哪个版本创建的：
SELECT DISTINCT version['created'], version['upgraded'] FROM "information_schema"."tables" WHERE table_schema not IN ('sys','information_schema','pg_catalog')
SELECT distinct min_lucene_version FROM sys.shards
如果只有较旧的 table 受到影响，我建议使用 4.2.7 重新索引那些 table，然后 运行 升级
SHOW CREATE table old;
INSERT INTO new SELECT * FROM old;
--or
COPY old TO;
COPY new FROM;