com.google.cloud.spanner.SpannerException: DEADLINE_EXCEEDED

Question

我已经被这个问题困扰了一天多了，我想这里有人可能知道答案。基本上，作为一个简单的测试，我试图从 table 中读取数据并将结果简单地输出到日志文件中。 table 相当大（约 1.67 亿行）。我不断收到以下错误

java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: com.google.cloud.spanner.SpannerException: DEADLINE_EXCEEDED: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 119997536405ns

接着是这个：

Workflow failed. Causes: S15:SpannerIO.ReadAll/Read from Cloud Spanner/Shuffle partitions/Reshuffle/GroupByKey/Read+SpannerIO.ReadAll/Read from Cloud Spanner/Shuffle partitions/Reshuffle/GroupByKey/GroupByWindow+SpannerIO.ReadAll/Read from Cloud Spanner/Shuffle partitions/Reshuffle/ExpandIterable+SpannerIO.ReadAll/Read from Cloud Spanner/Shuffle partitions/Values/Values/Map+SpannerIO.ReadAll/Read from Cloud Spanner/Read from Partitions+ParDo(FinishProcess) failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers: ricardocascadejob-m0771463-12021322-cpj6-harness-4nvn Root cause: Work item failed., ricardocascadejob-m0771463-12021322-cpj6-harness-c485 Root cause: Work item failed., ricardocascadejob-m0771463-12021322-cpj6-harness-kcgb Root cause: Work item failed., ricardocascadejob-m0771463-12021322-cpj6-harness-kcgb Root cause: Work item failed.

这是数据流运行上的主要代码

PipelineOptionsFactory.register(RicardoPriceLoadOptions.class);
    RicardoPriceLoadOptions opts = PipelineOptionsFactory.fromArgs(args)
        .withValidation().as(RicardoPriceLoadOptions.class);
    Pipeline pipeline = Pipeline.create(opts);

    SpannerConfig spannerConfig =
        SpannerConfig.create()
            .withProjectId(opts.getGcpProjectId())
            .withInstanceId(opts.getSpannerInstanceId())
            .withDatabaseId(opts.getSpannerDatabaseId());

    PCollectionView<Transaction> tx =
        pipeline.apply(SpannerIO.createTransaction().withSpannerConfig(spannerConfig));

    //Fetch All Price Events
    PCollection<Struct> pepLIst = pipeline.apply(Create.of(ReadOperation.create()
        .withColumns("DisabledFlag", "PriceEventPriceableId", "PriceableItemId",
            "OutgoingType", "PriceOriginal", "PriceIntermediate", "PriceRetail", "SaleValue", "SaleValueIntermediate",
            "SchedulableFlag", "SendToSiteFlag", "StartTime", "EndTime", "DisplayCode")
        .withTable("ABC")))
        .apply(SpannerIO.readAll().withTransaction(tx).withSpannerConfig(spannerConfig));

    pepLIst.apply(ParDo.of(new FinishProcessFn()));

最后一个 DoFn 很简单，它只记录扳手行。

public class FinishProcessFn extends DoFn<Struct, Void> {

@ProcessElement
public void process(@Element Struct elem) {
    log.debug(elem.toString());
}

}

我已尝试 google 此处显示的建议

Common Errors

代码看起来很简单，但我不确定为什么我总是收到上面的错误。感谢任何输入或帮助。

谢谢！

这是 table 架构

    CREATE TABLE ABC (
    PriceEventPriceableId INT64 NOT NULL,
    Created TIMESTAMP NOT NULL,
    CreatedBy STRING(MAX) NOT NULL,
    DisabledFlag STRING(MAX) NOT NULL,
    DisplayCode STRING(MAX),
    EndTime TIMESTAMP,
    ErrorCode INT64,
    EstablishmentOverrideFlag STRING(MAX),
    LastUpdated TIMESTAMP NOT NULL,
    LastUpdatedBy STRING(MAX) NOT NULL,
    NotApplicableFlag STRING(MAX),
    OnSaleRatioOverrideFlag STRING(MAX),
    OutgoingType INT64,
    OwnedValue STRING(MAX),
    ParentPriceableItemId INT64,
    PriceableItemId INT64 NOT NULL,
    PriceEventId INT64 NOT NULL,
    PriceIntermediate STRING(MAX),
    PriceOriginal STRING(MAX),
    PriceRetail STRING(MAX),
    ReasonUnschedulable STRING(MAX),
    SaleValue STRING(MAX),
    SaleValueIntermediate STRING(MAX),
    SavingsMaxOverrideFlag STRING(MAX),
    SchedulableFlag STRING(MAX),
    SendToSiteFlag STRING(MAX),
    SentToSiteDate DATE,
    StartTime TIMESTAMP,
    StoredValue STRING(MAX),
    TenPercentOverrideFlag STRING(MAX),
    Timestamp TIMESTAMP NOT NULL OPTIONS (allow_commit_timestamp=true),
) PRIMARY KEY (PriceEventPriceableId)

Answer 1

DEADLINE_EXCEEDED error表示操作没有在给定时间内完成。

对于更改系统状态的操作，即使操作已成功完成，也可能会返回此错误（例如，来自服务器的成功响应可能已延迟足够长的时间截止日期).

我在 Google Cloud Status Dashboard 中发现 Cloud Spanner 没有问题，但是，最好联系 GCP 支持，以便他们可以检查您的项目并采取更深入地研究这个问题。

Answer 2

因此，作为更新，我为解决该问题所做的工作是使用完全相同版本的光束和 google 扳手驱动程序作为 gpc 数据流中可用的 Spanner Export to Avro 模板，我的代码开始了神奇地工作。我没有做任何代码更改。

com.google.cloud.spanner.SpannerException: DEADLINE_EXCEEDED

com.google.cloud.spanner.SpannerException: DEADLINE_EXCEEDED

google-cloud-dataflow

apache-beam

google-cloud-spanner