在 Google 数据流上执行 Nexmark 基准测试套件

Question

我正在尝试运行 Google Dataflow 上的 Nexmark 基准测试套件，我阅读了 documentation，我想我知道一般设置。

但是，文档中缺少的一件事是初始设置，它似乎提供了很多运行ning 参数，但没有关于下载、编译和执行内容的实际命令。

如何开始基准测试过程？

编辑：

我下载了 Apache Beam 存储库并根据文档执行了以下命令：

./gradlew :sdks:java:testing:nexmark:run \
    -Pnexmark.runner=":runners:google-cloud-dataflow-java" \
    -Pnexmark.args="
        --runner=DataflowRunner
        --suite=SMOKE
        --streamTimeout=60
        --streaming=true
        --manageResources=false
        --monitorJobs=true
        --project=${PROJECT}
        --zone=${ZONE}
        --workerMachineType=n1-highmem-8
        --stagingLocation=${STAGING_LOCATION}
        --sourceType=PUBSUB
        --pubSubMode=PUBLISH_ONLY
        --pubsubTopic=${PUBSUB_TOPIC}
        --resourceNameMode=VERBATIM
        --manageResources=false
        --numEventGenerators=64
        --numWorkers=16
        --maxNumWorkers=16
        --firstEventRate=100000
        --nextEventRate=100000
        --ratePeriodSec=3600
        --isRateLimited=true
        --avgPersonByteSize=500
        --avgAuctionByteSize=500
        --avgBidByteSize=500
        --probDelayedEvent=0.000001
        --occasionalDelaySec=3600
        --numEvents=0
        --useWallclockEventTime=true
        --usePubsubPublishTime=true
        --experiments=enable_custom_pubsub_sink"

但是，我运行遇到了一个关于绑定的新问题，错误如下：

> Task :sdks:java:testing:nexmark:run FAILED
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/andy/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-jdk14/1.7.30/d35953dd2fe54ebe39fdf18cfd82fe6eb35b25ed/slf4j-jdk14-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/andy/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-log4j12/1.7.30/c21f55139d8141d2231214fb1feaf50a1edca95e/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.JDK14LoggerFactory]
Exception in thread "main" java.lang.IllegalArgumentException: expected one element but was: <false, false>
        at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators.getOnlyElement(Iterators.java:316)
        at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables.getOnlyElement(Iterables.java:254)
        at org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1936)
        at org.apache.beam.sdk.options.PipelineOptionsFactory.access0(PipelineOptionsFactory.java:144)
        at org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.as(PipelineOptionsFactory.java:327)
        at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:106)
        at org.apache.beam.sdk.nexmark.Main.main(Main.java:467)

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':sdks:java:testing:nexmark:run'.
> Process 'command '/usr/lib/jvm/java-11-openjdk/bin/java'' finished with non-zero exit value 1

Answer 1

要进行设置，您必须下载 apache/beam 存储库。

转到根目录，然后按照documentation中的示例执行Gradle任务。

在 Google 数据流上执行 Nexmark 基准测试套件

Executing Nexmark benchmark suite on Google Dataflow

google-cloud-dataflow

apache-beam