DCOS Kafka 部署卡在 Vagrant 集群上

Question

我正在使用最新的 DCOS 版本 (0.8.0) 并启动了一个本地 vagrant 集群，其中包含 4 个私有代理节点、一个 public 代理节点、一个主节点和一个引导机器

vagrant up m1 a1 a2 a3 a4 p1 boot

然后我使用以下命令安装了具有默认配置的 kafka：

dcos package install kafka

检查马拉松日志后，mesos 似乎正在拒绝为 kafka 提供的资源，因此无法部署所有三个代理。我怀疑问题出在资源提供的角色与所需的 kafka 角色之间不匹配。

聚合集群统计信息： CPU：18股内存：7GiB

这是马拉松日志的快照：

s.offer.OfferEvaluator: Satisfying resource requirement: name: "cpus" type: SCALAR scalar { value: 1.0 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "" } } }
with resource: name: "cpus" type: SCALAR scalar { value: 1.0 } role: "*"
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Reserves Resource
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Fulfilled resource: name: "cpus" type: SCALAR scalar { value: 1.0 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "f8468a5a-8166-4d5d-a594-f197a8b356cd" } } }
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.MesosResourcePool: Retrieving resource for reservation
WARN  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Failed to satisfy resource requirement: name: "mem" type: SCALAR scalar { value: 2304.0 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "" } } }
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.MesosResourcePool: Retrieving resource for reservation
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Satisfying resource requirement: name: "cpus" type: SCALAR scalar { value: 0.5 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "" } } }
with resource: name: "cpus" type: SCALAR scalar { value: 0.5 } role: "*"
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Reserves Resource
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Fulfilled resource: name: "cpus" type: SCALAR scalar { value: 0.5 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "37ba47a3-7c6e-47a2-972c-d35d130ae8df" } } }
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.MesosResourcePool: Retrieving resource for reservation
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Satisfying resource requirement: name: "mem" type: SCALAR scalar { value: 256.0 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "" } } }
with resource: name: "mem" type: SCALAR scalar { value: 256.0 } role: "*"
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Reserves Resource
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Fulfilled resource: name: "mem" type: SCALAR scalar { value: 256.0 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "8f937e43-c80a-4434-a858-08e6bdb02cc2" } } }
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.MesosResourcePool: Retrieving resource for reservation
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Satisfying resource requirement: name: "cpus" type: SCALAR scalar { value: 1.0 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "" } } }
with resource: name: "cpus" type: SCALAR scalar { value: 1.0 } role: "*"
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Reserves Resource
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Fulfilled resource: name: "cpus" type: SCALAR scalar { value: 1.0 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "b568cfaa-c216-4479-9853-5bc4e6ee2d9e" } } }
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.MesosResourcePool: Retrieving resource for reservation
WARN  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Failed to satisfy resource requirement: name: "mem" type: SCALAR scalar { value: 2304.0 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "" } } }
WARN  [2016-08-02 16:28:08,919] org.apache.mesos.scheduler.plan.DefaultStageScheduler: Unable to find any offers which fulfill requirement provided by block broker-0: org.apache.mesos.offer.OfferRequirement@2a28d9e[avoidAgents=[],colocateAgents=[],taskRequirements=[org.apache.mesos.offer.TaskRequirement@5c8e5b0f],executorRequirement=org.apache.mesos.offer.ExecutorRequirement@6179902f[executorInfo=executor_id {
  value: ""
}
resources {
  name: "cpus"
  type: SCALAR
  scalar {
    value: 0.5
  }
  role: "kafka-role"
  reservation {
    principal: "kafka-principal"
    labels {
      labels {
        key: "resource_id"
        value: ""
      }
    }
  }
}
resources {
  name: "mem"
  type: SCALAR
  scalar {
    value: 256.0
  }
  role: "kafka-role"
  reservation {
    principal: "kafka-principal"
    labels {
      labels {
        key: "resource_id"
        value: ""
      }
    }
  }
}
command {
  uris {
    value: "https://downloads.mesosphere.com/kafka/assets/jre-8u91-linux-x64.tar.gz"
  }
  uris {
    value: "https://downloads.mesosphere.com/kafka/assets/kafka_2.11-0.10.0.0.tgz"
  }
  uris {
    value: "https://downloads.mesosphere.com/kafka/assets/1.1.9-0.10.0.0/overrider.zip"
  }
  uris {
    value: "https://downloads.mesosphere.com/kafka/assets/1.1.9-0.10.0.0/executor.zip"
  }
  environment {
    variables {
      name: "JAVA_HOME"
      value: "jre1.8.0_91"
    }
  }
  value: "./executor/bin/kafka-executor -Dlogback.configurationFile=executor/conf/logback.xml"
}
framework_id {
  value: "2601d122-2b59-4823-83f8-84351376d53c-0001"
}
name: "broker-0"
,resourceRequirements=[org.apache.mesos.offer.ResourceRequirement@39ba5849, org.apache.mesos.offer.ResourceRequirement@2f62ec9d]]]
INFO  [2016-08-02 16:28:08,919] com.mesosphere.dcos.kafka.plan.KafkaUpdateBlock: broker-0: changed status from: Pending to: Pending
INFO  [2016-08-02 16:28:08,920] com.mesosphere.dcos.kafka.scheduler.KafkaRepairScheduler: Missing brokerIds: []
INFO  [2016-08-02 16:28:08,921] org.apache.mesos.offer.ResourceCleaner: 4 offers with 16 resources => 0 destroy and 0 unreserve operations
INFO  [2016-08-02 16:28:08,921] com.mesosphere.dcos.kafka.scheduler.KafkaScheduler: Accepted 0 of 4 offers: []
INFO  [2016-08-02 16:28:08,921] com.mesosphere.dcos.kafka.scheduler.KafkaScheduler: Declining offer: 2601d122-2b59-4823-83f8-84351376d53c-O1343
INFO  [2016-08-02 16:28:08,921] com.mesosphere.dcos.kafka.scheduler.KafkaScheduler: Declining offer: 2601d122-2b59-4823-83f8-84351376d53c-O1344
INFO  [2016-08-02 16:28:08,921] com.mesosphere.dcos.kafka.scheduler.KafkaScheduler: Declining offer: 2601d122-2b59-4823-83f8-84351376d53c-O1345
INFO  [2016-08-02 16:28:08,921] com.mesosphere.dcos.kafka.scheduler.KafkaScheduler: Declining offer: 2601d122-2b59-4823-83f8-84351376d53c-O1346
INFO  [2016-08-02 16:28:14,928]

Answer 1

DCOS 1.8中的Kafka框架是一个有状态的服务。它专门设计用于为每个代理分配存储并在代理实例和该存储之间维护 link（因此 broker-0 不会重新部署到没有最初为 broker- 分配存储的主机） 0).我的经验是，当不满足存储限制时，围绕资源提供的消息不是很清楚。默认情况下，程序包将为每个代理请求 5 GB 的专用存储空间。

我对 vagrant 没有太多经验，但可能是 vagrant 没有为 DC/OS 公开必要的 "storage availability" 来满足代理资源请求。

Answer 2

问题确实如日志所报告的那样：内存不足，无法使用默认设置调度 kafka。我能够使用默认的 vagrant 配置和使用以下 kafka 配置的三个代理节点在 DCOS vagrant 上部署 kafka：

{
    "brokers": {
        "count": 1,
        "mem": 512,
        "disk": 1000
    }
}

DCOS Kafka 部署卡在 Vagrant 集群上

DCOS Kafka Deployment Stuck on Vagrant Cluster

vagrant

apache-kafka

mesos

dcos