org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.AvroRuntimeException: 未知数据类型

Question

我使用的是 avro 版本 1.7.7，scala 版本 2.11.8

我的 avro 架构称为 EntityAuth，其中的枚举如下所示

            {
              "name": "reason",
              "type": [
                "null", 
                {
                  "type": "enum",
                  "name": "BlockedSiteReasonType",
                  "symbols": [
                    "REASON_1",
                    "REASON_2",
                    "REASON_3",
                    "REASON_4",
                    "ADMIN_SITE_CATEGORY_FILTER",
                    "REASON_6",
                    "REASON_7",
                    "REASON_8",
                    "REASON_9"
                  ]
                }
              ],
              "doc": "Agent blocked the entity",
              "default": null
            }

我编写 avro 文件的代码如下所示。 EntityAuth 是 avro 架构 class

class EntitySerializationStream(codec: CodecFactory = CodecFactory.snappyCodec()) {
  private val schema = EntityAuth.getClassSchema
  private val datumWriter = new GenericDatumWriter[EntityAuth](schema)
  private val fileName = "some-file-name"
  private val tempFile = File.createTempFile(fileName, "avro.tmp")
  tempFile.deleteOnExit()

  private val dataFileWriter = new DataFileWriter[EntityAuth](datumWriter)
  dataFileWriter.setCodec(codec)
  dataFileWriter.create(schema, tempFile)

  def append(e: EntityAuth) = {
    dataFileWriter.append(e)
  }

当我尝试将记录附加到通用数据文件编写器时出现的异常是

org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.AvroRuntimeException: Unknown datum type com.goguardian.events.auth.BlockedSiteReasonType: ADMIN_SITE_CATEGORY_FILTER
        at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:296)
Caused by: org.apache.avro.AvroRuntimeException: Unknown datum type com.goguardian.events.auth.BlockedSiteReasonType: ADMIN_SITE_CATEGORY_FILTER
        at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:636)
        at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:601)
        at org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
        at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
        at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
        at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:138)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:68)
        at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
        at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
        at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290)
        ... 22 more

这仅在我编写 avro 文件时发生。当我读取 avro 文件时，我可以很好地解析它。

Answer 1

如果您使用此模式生成 Java Class。请替换

private val schema = EntityAuth.getClassSchema
private val datumWriter = new GenericDatumWriter[EntityAuth](schema)

至

private val datumWriter = new SpecificDatumWriter[EntityAuth](classOf[EntityAuth])

SpecificDatumWriter 用于生成 Java 类.
GenericDatumWriter 适用于通用 Java 对象，没有代码生成。

org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.AvroRuntimeException: 未知数据类型

org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.AvroRuntimeException: Unknown datum type

enums

scala

avro