为什么 kafka-avro-console-producer 不遵守该字段的默认值？

Question

尽管为字段定义了默认值，kafka-avro-console-producer 完全忽略它：

$ kafka-avro-console-producer --broker-list localhost:9092 --topic test-avro \
--property schema.registry.url=http://localhost:8081 --property \
value.schema='{"type":"record","name":"myrecord1","fields": \
[{"name":"f1","type":"string"},{"name": "f2", "type": "int", "default": 0}]}'

{"f1": "value1"}

org.apache.kafka.common.errors.SerializationException: Error 
deserializing json {"f1": "value1"} to Avro of schema 
{"type":"record","name":"myrecord1","fields": 
[{"name":"f1","type":"string"},{"name":"f2","type":"int","default":0}]}
Caused by: org.apache.avro.AvroTypeException: Expected int. Got END_OBJECT
    at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:698)
    at org.apache.avro.io.JsonDecoder.readInt(JsonDecoder.java:172)
    at org.apache.avro.io.ValidatingDecoder.readInt(ValidatingDecoder.java:83)
    at org.apache.avro.generic.GenericDatumReader.readInt(GenericDatumReader.java:511)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:182)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152)
    at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:240)
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:230)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:174)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:144)
    at io.confluent.kafka.formatter.AvroMessageReader.jsonToAvro(AvroMessageReader.java:213)
    at io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:180)
    at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:54)
    at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)

如何使用它才能接受默认值？顶级配置设置为 "BACKWARD" 兼容性级别检查，但我认为这与问题无关。此架构是版本 2，版本 1 仅使用 f1 字段定义，但正如我所说，我认为这无关紧要。

Answer 1

该错误表明消息与您定义的 Avro 架构不兼容。据我了解，您希望字段 f2 允许 null 值。为此，您需要将 value.schema 更改为（注意 "type" 的定义）：

value.schema='{"type":"record","name":"myrecord1","fields": [{"name":"f1","type":"string"},{"name": "f2", "type": ["null", "int"], "default": 0}]}'

但您仍然需要使用空值定义 f2 键。以下应该为您解决问题：

kafka-avro-console-producer --broker-list localhost:9092 --topic test-avro \ 
    --property schema.registry.url=http://localhost:8081 \ 
    --property value.schema='{"type":"record","name":"myrecord1","fields": [{"name":"f1","type":"string"},{"name": "f2", "type": ["null", "int"], "default": 0}]}'

{"f1":"value1","f2":null}

并且您可以使用 kafka-avro-console-consumer 确认这是否有效：

kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic test-avro --from-beginning
{"f1":"value1","f2":null}
^CProcessed a total of 1 messages

Answer 2

定义在Avro spec

default: A default value for this field, used when reading instances that lack this field

因此，生产者仍需提供该字段。

我不确定在使用 Avro 控制台生成器时是否可以完全排除某个字段，因为即使您像 Giorgos 显示的那样使该字段可以为空，您仍然需要明确设置它。

为什么 kafka-avro-console-producer 不遵守该字段的默认值？

Why kafka-avro-console-producer doesn't honour the default value for the field?

avro

apache-kafka

kafka-producer-api

confluent-schema-registry