部署在 DC/OS (Mesos) 中时，Spark Kafka 流作业未发现协调器
Spark Kafka streaming job doesn't discover coordinator when deployed in DC/OS (Mesos)
我在 java 中按照 https://spark.apache.org/docs/2.1.0/streaming-kafka-0-10-integration.html 中的说明实施了一个火花流作业。当 Kafka 和 Cassandra 服务器独立时，它可以完美运行。日志提取
...
value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
18/02/04 20:40:35 INFO AppInfoParser: Kafka version : 0.11.0.0
18/02/04 20:40:35 INFO AppInfoParser: Kafka commitId : cb8625948210849f
18/02/04 20:40:35 INFO AbstractCoordinator: Discovered coordinator spark-vm:9092 (id: 2147483647 rack: null) for group group_1.
18/02/04 20:40:35 INFO ConsumerCoordinator: Revoking previously assigned partitions [] for group group_1
18/02/04 20:40:35 INFO AbstractCoordinator: (Re-)joining group group_1
[2018-02-04 20:40:35,929] INFO [GroupCoordinator 0]: Preparing to rebalance group group_1 with old generation 6 (__consumer_offsets-49) (kafka.coordinator.group.GroupCoordinator)
[2018-02-04 20:40:35,946] INFO [GroupCoordinator 0]: Stabilized group group_1 generation 7 (__consumer_offsets-49) (kafka.coordinator.group.GroupCoordinator)
...
当我在 mesos 集群中部署时出现问题，因为作业连接到 kafka 但它没有发现任何协调器
metrics.sample.window.ms = 30000
auto.offset.reset = latest
18/02/04 19:27:41 INFO AppInfoParser: Kafka version : 0.10.0.1
18/02/04 19:27:41 INFO AppInfoParser: Kafka commitId : a7a17cdec9eaa6c5
18/02/04 19:27:42 INFO MesosCoarseGrainedSchedulerBackend: Mesos task 1 is now TASK_RUNNING
18/02/04 19:27:44 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.65.111:50842) with ID 1
18/02/04 19:27:45 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.65.111:39485 with 366.3 MB RAM, BlockManagerId(1, 192.168.65.111, 39485, None)
18/02/04 19:27:47 INFO CassandraConnector: Disconnected from Cassandra cluster: cassandra
2018-02-04 19:32:19,161:7(0x7fb4fd02b700):ZOO_WARN@zookeeper_interest@1570: Exceeded deadline by 12ms
Kafka 使用的 ZK 是 master.mesos:2181/dcos-service-kafka 但我不知道如何在 Spark 作业中配置它，我什至不确定这是问题还是只是kafka 版本问题。
第一个问题是连接问题。 Spark 作业无法使用 DC/OS 提供的 VIP 端点连接到 Kafka，但日志没有像这些技术的新手所期望的那样抛出任何异常。
我在 java 中按照 https://spark.apache.org/docs/2.1.0/streaming-kafka-0-10-integration.html 中的说明实施了一个火花流作业。当 Kafka 和 Cassandra 服务器独立时，它可以完美运行。日志提取
...
value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
18/02/04 20:40:35 INFO AppInfoParser: Kafka version : 0.11.0.0
18/02/04 20:40:35 INFO AppInfoParser: Kafka commitId : cb8625948210849f
18/02/04 20:40:35 INFO AbstractCoordinator: Discovered coordinator spark-vm:9092 (id: 2147483647 rack: null) for group group_1.
18/02/04 20:40:35 INFO ConsumerCoordinator: Revoking previously assigned partitions [] for group group_1
18/02/04 20:40:35 INFO AbstractCoordinator: (Re-)joining group group_1
[2018-02-04 20:40:35,929] INFO [GroupCoordinator 0]: Preparing to rebalance group group_1 with old generation 6 (__consumer_offsets-49) (kafka.coordinator.group.GroupCoordinator)
[2018-02-04 20:40:35,946] INFO [GroupCoordinator 0]: Stabilized group group_1 generation 7 (__consumer_offsets-49) (kafka.coordinator.group.GroupCoordinator)
...
当我在 mesos 集群中部署时出现问题，因为作业连接到 kafka 但它没有发现任何协调器
metrics.sample.window.ms = 30000
auto.offset.reset = latest
18/02/04 19:27:41 INFO AppInfoParser: Kafka version : 0.10.0.1
18/02/04 19:27:41 INFO AppInfoParser: Kafka commitId : a7a17cdec9eaa6c5
18/02/04 19:27:42 INFO MesosCoarseGrainedSchedulerBackend: Mesos task 1 is now TASK_RUNNING
18/02/04 19:27:44 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.65.111:50842) with ID 1
18/02/04 19:27:45 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.65.111:39485 with 366.3 MB RAM, BlockManagerId(1, 192.168.65.111, 39485, None)
18/02/04 19:27:47 INFO CassandraConnector: Disconnected from Cassandra cluster: cassandra
2018-02-04 19:32:19,161:7(0x7fb4fd02b700):ZOO_WARN@zookeeper_interest@1570: Exceeded deadline by 12ms
Kafka 使用的 ZK 是 master.mesos:2181/dcos-service-kafka 但我不知道如何在 Spark 作业中配置它，我什至不确定这是问题还是只是kafka 版本问题。
第一个问题是连接问题。 Spark 作业无法使用 DC/OS 提供的 VIP 端点连接到 Kafka，但日志没有像这些技术的新手所期望的那样抛出任何异常。