无法通过 Env 变量设置带有 camelCase 名称的主题
Unable to set topic with camelCase name via Env variable
我正在使用 helm 部署 zeebe。使用 extraInitContainers 指令,我设法包含 kafka-exporter 3.1.1 并正确加载。
在yml文件中我设置了一系列env变量,下面你可以看到一个摘录:
env:
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_PRODUCER_SERVERS
value: acme-infrastructure-kafka:9092
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_PRODUCER_CONFIG
value: "linger.ms=5\nbuffer.memory=8388608\nbatch.size=32768\nmax.block.ms=5000"
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_CLASSNAME
value: io.zeebe.exporters.kafka.KafkaExporter
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_JARPATH
value: /usr/local/zeebe/exporters/zeebe-kafka-exporter.jar
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_RECORDS_DEFAULTS_TYPE
value: ""
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_RECORDS_DEFAULTS_TOPIC
value: "zeebe"
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_RECORDS_JOB_TYPE
value: "event"
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_RECORDS_JOB_TOPIC
value: "zeebe-job"
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_RECORDS_PROCESSINSTANCE_TYPE
value: "event"
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_RECORDS_PROCESSINSTANCE_TOPIC
value: "zeebe-process-instance"
使用此配置,我没有收到任何 zeebe 主题记录,但收到了类型为 EVENT 的 zeebe-job 和 zeebe-process-instance 的主题,如文档所述。但我只得到工作,我认为流程实例主题不起作用,因为它想要这个: json 配置文件中的 processInstance 但它得到这个 处理实例。
从zeebe logs,开机可以看到配置,主题全部小写:
io.camunda.zeebe.broker.exporter - Configured Kafka exporter: Config{producer=ProducerConfig{clientId='zeebe', closeTimeout=PT20S, config={batch.size=32768, max.block.ms=5000, buffer.memory=8388608, linger.ms=5}, requestTimeout=PT5S, maxBlockingTimeout=PT2S, servers=[datamanent-infrastructure-kafka:9092]}, records=RecordsConfig{typeMap={JOB=RecordConfig{allowedTypes=[EVENT], topic='zeebe-job'}, INCIDENT=RecordConfig{allowedTypes=[EVENT], topic='zeebe-incident'}, ERROR=RecordConfig{allowedTypes=[EVENT], topic='zeebe-error'}, PROCESS=RecordConfig{allowedTypes=[EVENT], topic='zeebe-process'}}, defaults=RecordConfig{allowedTypes=[], topic='zeebe'}}, maxBatchSize=100, commitInterval=PT1S}
2022-03-21 14:16:29.038 [] [main] INFO
io.camunda.zeebe.broker.system - Version: 1.3.4
2022-03-21 14:16:29.068 [] [main] INFO
io.camunda.zeebe.broker.system - Starting broker 0 with configuration {
"kafka" : {
"jarPath" : "/usr/local/zeebe/exporters/zeebe-kafka-exporter.jar",
"className" : "io.zeebe.exporters.kafka.KafkaExporter",
"args" : {
"maxbatchsize" : "100",
"producer" : {
"clientid" : "zeebe",
"servers" : "acme-infrastructure-kafka:9092",
"config" : "linger.ms=5\nbuffer.memory=8388608\nbatch.size=32768\nmax.block.ms=5000"
},
"records" : {
"job" : {
"topic" : "zeebe-job",
"type" : "event"
},
"process" : {
"type" : "event",
"topic" : "zeebe-process"
},
"defaults" : {
"type" : "",
"topic" : "zeebe"
},
"error" : {
"topic" : "zeebe-error",
"type" : "event"
},
"incident" : {
"type" : "event",
"topic" : "zeebe-incident"
},
"processinstance" : {
"topic" : "zeebe-process-instance",
"type" : "event"
}
},
"maxblockingtimeoutms" : "1000",
"flushintervalms" : "1000"
},
"external" : true
}
这里是社区项目的link:
https://github.com/camunda-community-hub/zeebe-kafka-exporter
这是一个已知限制https://github.com/camunda/zeebe/issues/4724 we hope we can fix this with use of Spring's ApplicationContext to instantiate exporter arguments see https://github.com/camunda/zeebe/issues/7628
作为正确配置的解决方法,可以使用 JAVA_OPTIONS
环境变量:
所以例如写 ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_RECORDS_PROCESSINSTANCE_TYPE=event
为 JAVA_OPTS="${JAVA_OPTS} -Dzeebe.broker.exporters.kafka.args.records.processInstance.type=event
或者您可以通过 application.yaml https://github.com/camunda/zeebe/blob/main/dist/src/main/config/broker.standalone.yaml.template 来实现,您可以将其从 configmap 挂载到容器中。
我正在使用 helm 部署 zeebe。使用 extraInitContainers 指令,我设法包含 kafka-exporter 3.1.1 并正确加载。 在yml文件中我设置了一系列env变量,下面你可以看到一个摘录:
env:
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_PRODUCER_SERVERS
value: acme-infrastructure-kafka:9092
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_PRODUCER_CONFIG
value: "linger.ms=5\nbuffer.memory=8388608\nbatch.size=32768\nmax.block.ms=5000"
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_CLASSNAME
value: io.zeebe.exporters.kafka.KafkaExporter
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_JARPATH
value: /usr/local/zeebe/exporters/zeebe-kafka-exporter.jar
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_RECORDS_DEFAULTS_TYPE
value: ""
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_RECORDS_DEFAULTS_TOPIC
value: "zeebe"
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_RECORDS_JOB_TYPE
value: "event"
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_RECORDS_JOB_TOPIC
value: "zeebe-job"
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_RECORDS_PROCESSINSTANCE_TYPE
value: "event"
- name: ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_RECORDS_PROCESSINSTANCE_TOPIC
value: "zeebe-process-instance"
使用此配置,我没有收到任何 zeebe 主题记录,但收到了类型为 EVENT 的 zeebe-job 和 zeebe-process-instance 的主题,如文档所述。但我只得到工作,我认为流程实例主题不起作用,因为它想要这个: json 配置文件中的 processInstance 但它得到这个 处理实例。 从zeebe logs,开机可以看到配置,主题全部小写:
io.camunda.zeebe.broker.exporter - Configured Kafka exporter: Config{producer=ProducerConfig{clientId='zeebe', closeTimeout=PT20S, config={batch.size=32768, max.block.ms=5000, buffer.memory=8388608, linger.ms=5}, requestTimeout=PT5S, maxBlockingTimeout=PT2S, servers=[datamanent-infrastructure-kafka:9092]}, records=RecordsConfig{typeMap={JOB=RecordConfig{allowedTypes=[EVENT], topic='zeebe-job'}, INCIDENT=RecordConfig{allowedTypes=[EVENT], topic='zeebe-incident'}, ERROR=RecordConfig{allowedTypes=[EVENT], topic='zeebe-error'}, PROCESS=RecordConfig{allowedTypes=[EVENT], topic='zeebe-process'}}, defaults=RecordConfig{allowedTypes=[], topic='zeebe'}}, maxBatchSize=100, commitInterval=PT1S}
2022-03-21 14:16:29.038 [] [main] INFO
io.camunda.zeebe.broker.system - Version: 1.3.4
2022-03-21 14:16:29.068 [] [main] INFO
io.camunda.zeebe.broker.system - Starting broker 0 with configuration {
"kafka" : {
"jarPath" : "/usr/local/zeebe/exporters/zeebe-kafka-exporter.jar",
"className" : "io.zeebe.exporters.kafka.KafkaExporter",
"args" : {
"maxbatchsize" : "100",
"producer" : {
"clientid" : "zeebe",
"servers" : "acme-infrastructure-kafka:9092",
"config" : "linger.ms=5\nbuffer.memory=8388608\nbatch.size=32768\nmax.block.ms=5000"
},
"records" : {
"job" : {
"topic" : "zeebe-job",
"type" : "event"
},
"process" : {
"type" : "event",
"topic" : "zeebe-process"
},
"defaults" : {
"type" : "",
"topic" : "zeebe"
},
"error" : {
"topic" : "zeebe-error",
"type" : "event"
},
"incident" : {
"type" : "event",
"topic" : "zeebe-incident"
},
"processinstance" : {
"topic" : "zeebe-process-instance",
"type" : "event"
}
},
"maxblockingtimeoutms" : "1000",
"flushintervalms" : "1000"
},
"external" : true
}
这里是社区项目的link: https://github.com/camunda-community-hub/zeebe-kafka-exporter
这是一个已知限制https://github.com/camunda/zeebe/issues/4724 we hope we can fix this with use of Spring's ApplicationContext to instantiate exporter arguments see https://github.com/camunda/zeebe/issues/7628
作为正确配置的解决方法,可以使用 JAVA_OPTIONS
环境变量:
所以例如写 ZEEBE_BROKER_EXPORTERS_KAFKA_ARGS_RECORDS_PROCESSINSTANCE_TYPE=event
为 JAVA_OPTS="${JAVA_OPTS} -Dzeebe.broker.exporters.kafka.args.records.processInstance.type=event
或者您可以通过 application.yaml https://github.com/camunda/zeebe/blob/main/dist/src/main/config/broker.standalone.yaml.template 来实现,您可以将其从 configmap 挂载到容器中。