星火:配置文件'metrics.properties'

Spark: configuration file 'metrics.properties'

Spark 的配置文件(用于检索指标),即 /conf/metrics.properties,说明如下:

Within an instance, a "source" specifies a particular set of grouped metrics. there are two kinds of sources:

  1. Spark internal sources, like MasterSource, WorkerSource, etc, which will collect a Spark component's internal state. Each instance is paired with a Spark source that is added automatically.

  2. Common sources, like JvmSource, which will collect low level state. These can be added through configuration options and are then loaded using reflection.

下面是一些例子,如

master.source.jvm.class=org.apache.spark.metrics.source.JvmSource

然而,没有示例解释如何从 MasterSourceWorkerSource 等其他来源获取指标,仅替换 JvmSource 并不能解决问题。

建议?

MasterSource 或 WorkerSource -- 都是内部源 -- 由指标自动跟踪,因此无需明确指定。

例如,

master.sink.cvs.class=org.apache.spark.metrics.sink.CsvSink

将每 10 秒将主实例指标输出到 /tmp 目录。