Kafka 连接 OCI Stream 服务失败
Kafka connect failed with OCI Stream service
我正在尝试设置 Kafka 与 OCI 流服务的连接。
下面是Kafka连接docker配置,
connect:
image: confluentinc/cp-kafka-connect-base:7.0.0
hostname: connect
ports:
- 8083:8083
container_name: connect
environment:
CONNECT_BOOTSTRAP_SERVERS: '<BOOTSTRAP_SERVER>'
CONNECT_REST_ADVERTISED_HOST_NAME: connect
CONNECT_REST_PORT: 8083
CONNECT_GROUP_ID: compose-connect-group
CONNECT_CONFIG_STORAGE_TOPIC: '<KAFKA_CONNECT_OCI_ID>'-config
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 3
CONNECT_OFFSET_FLUSH_INTERVAL_MS: 200000
CONNECT_OFFSET_STORAGE_TOPIC: '<KAFKA_CONNECT_OCI_ID>'-offset
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 3
CONNECT_STATUS_STORAGE_TOPIC: '<KAFKA_CONNECT_OCI_ID>'-status
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 3
CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
CONNECT_LOG4J_ROOT_LOGLEVEL: "INFO"
CONNECT_LOG4J_LOGGERS: "org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR,com.mongodb.kafka=DEBUG"
CONNECT_PLUGIN_PATH: /usr/share/confluent-hub-components
CONNECT_SECURITY_PROTOCOL: SASL_SSL
CONNECT_SASL_MECHANISM: PLAIN
CONNECT_SASL_JAAS_CONFIG: "<JASS_CONFIG_AS_IN_STREAMINGPOOL_SETTINGS>"
CONNECT_PRODUCER_SECURITY_PROTOCOL: SASL_SSL
CONNECT_PRODUCER_SASL_MECHANISM: PLAIN
CONNECT_PRODUCER_SASL_JAAS_CONFIG: "<JASS_CONFIG_AS_IN_STREAMINGPOOL_SETTINGS>"
CONNECT_CONSUMER_SECURITY_PROTOCOL: SASL_SSL
CONNECT_CONSUMER_SASL_MECHANISM: PLAIN
CONNECT_CONSUMER_SASL_JAAS_CONFIG: "<JASS_CONFIG_AS_IN_STREAMINGPOOL_SETTINGS>"
CONNECT_CONSUMER_MAX_POLL_RECORDS: 1000
KAFKA_HEAP_OPTS: "-Xms1G -Xmx4G"
低于错误
ERROR [Worker clientId=connect-1, groupId=compose-connect-group] Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder) org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before the position for partition <KAFKA_CONNECT_OCI_ID>-offset-0 could be determined
INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect)
我不认为存在网络问题,因为当我有不同的主题名称或错误的 bootstrap 服务器时,我会得到不同的错误,甚至无法启动。但对我来说,这个错误在一分钟后出现,平均时间 REST 调用 GET 请求工作。
请注意,AWS MSK 的类似配置适用于我。
请帮我解决问题。
更新: OCI 流媒体服务不支持最新版本的 Kafka 连接。在我改成 5.x
后它起作用了
OCI 流媒体服务不支持最新版本的 Kafka 连接。在我改成 5.x
后它起作用了
我正在尝试设置 Kafka 与 OCI 流服务的连接。
下面是Kafka连接docker配置,
connect:
image: confluentinc/cp-kafka-connect-base:7.0.0
hostname: connect
ports:
- 8083:8083
container_name: connect
environment:
CONNECT_BOOTSTRAP_SERVERS: '<BOOTSTRAP_SERVER>'
CONNECT_REST_ADVERTISED_HOST_NAME: connect
CONNECT_REST_PORT: 8083
CONNECT_GROUP_ID: compose-connect-group
CONNECT_CONFIG_STORAGE_TOPIC: '<KAFKA_CONNECT_OCI_ID>'-config
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 3
CONNECT_OFFSET_FLUSH_INTERVAL_MS: 200000
CONNECT_OFFSET_STORAGE_TOPIC: '<KAFKA_CONNECT_OCI_ID>'-offset
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 3
CONNECT_STATUS_STORAGE_TOPIC: '<KAFKA_CONNECT_OCI_ID>'-status
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 3
CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
CONNECT_LOG4J_ROOT_LOGLEVEL: "INFO"
CONNECT_LOG4J_LOGGERS: "org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR,com.mongodb.kafka=DEBUG"
CONNECT_PLUGIN_PATH: /usr/share/confluent-hub-components
CONNECT_SECURITY_PROTOCOL: SASL_SSL
CONNECT_SASL_MECHANISM: PLAIN
CONNECT_SASL_JAAS_CONFIG: "<JASS_CONFIG_AS_IN_STREAMINGPOOL_SETTINGS>"
CONNECT_PRODUCER_SECURITY_PROTOCOL: SASL_SSL
CONNECT_PRODUCER_SASL_MECHANISM: PLAIN
CONNECT_PRODUCER_SASL_JAAS_CONFIG: "<JASS_CONFIG_AS_IN_STREAMINGPOOL_SETTINGS>"
CONNECT_CONSUMER_SECURITY_PROTOCOL: SASL_SSL
CONNECT_CONSUMER_SASL_MECHANISM: PLAIN
CONNECT_CONSUMER_SASL_JAAS_CONFIG: "<JASS_CONFIG_AS_IN_STREAMINGPOOL_SETTINGS>"
CONNECT_CONSUMER_MAX_POLL_RECORDS: 1000
KAFKA_HEAP_OPTS: "-Xms1G -Xmx4G"
低于错误
ERROR [Worker clientId=connect-1, groupId=compose-connect-group] Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder) org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before the position for partition <KAFKA_CONNECT_OCI_ID>-offset-0 could be determined
INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect)
我不认为存在网络问题,因为当我有不同的主题名称或错误的 bootstrap 服务器时,我会得到不同的错误,甚至无法启动。但对我来说,这个错误在一分钟后出现,平均时间 REST 调用 GET 请求工作。
请注意,AWS MSK 的类似配置适用于我。
请帮我解决问题。
更新: OCI 流媒体服务不支持最新版本的 Kafka 连接。在我改成 5.x
后它起作用了OCI 流媒体服务不支持最新版本的 Kafka 连接。在我改成 5.x
后它起作用了