无法使用 Dataflow REST 传递作业参数 API

Question

我正在尝试通过 REST API 调用数据流模板，但是，当在正文中指定参数时，出现 INVALID_ARGUMENT 错误抛出。删除参数字段后，它可以正常工作。我已经尝试了 Javascript 和 Python 但得到了完全相同的错误。

from googleapiclient.discovery import build
from oauth2client.client import GoogleCredentials

def invoke_dataflow_job():
    credentials = GoogleCredentials.get_application_default()
    service = build('dataflow', 'v1b3', credentials=credentials)

    TEMPLATE_LOCATION = 'gs://xxx';
    PROJECT_ID = 'xxx';
    JOB_NAME = 'xxx';
    LOCATION = 'europe-west1';
    ZONE = 'europe-west1-b';
    TEMP_LOCATION = 'xxx';

    BODY = {
        'jobName': JOB_NAME,
        'gcsPath': TEMPLATE_LOCATION,
        'parameters': {
           'messageToPrint': 'Blue sky'
        },
        'environment': {
           'tempLocation': TEMP_LOCATION,
           'zone': ZONE,
        }
    }

    request = service.projects().locations().templates().create(projectId=PROJECT_ID, location=LOCATION, body=BODY)
    response = request.execute()
    return response

RESTAPI支持传参吗？查看example代码，我认为应该如此。

有什么建议吗？

错误信息：

The workflow could not be created. Causes: (4adc3dbfd180d8a2): Found unexpected parameters: ['messageToPrint' (perhaps you meant 'userAgent')]

更新 1：

DoFn 示例：

public static class BuildPathFn extends DoFn<String, Void> {
    private static final long serialVersionUID = -2815123956194177539L;
    private final ValueProvider<String> baseDir; 
    private final ValueProvider<String> year; 
    private final ValueProvider<String> month; 
    private final ValueProvider<String> day; 
    private final ValueProvider<String> hour; 
    private final ValueProvider<String> filePattern; 

    public BuildPathFn (ValueProvider<String> baseDir, ValueProvider<String> year, 
                            ValueProvider<String> month, ValueProvider<String> day, 
                            ValueProvider<String> hour, ValueProvider<String> filePattern) {
        this.baseDir = baseDir;
        this.year = year;
        this.month = month;
        this.day = day;
        this.hour = hour;
        this.filePattern = filePattern;
    }

    @ProcessElement
    public void processElement(@Element String element, OutputReceiver<Void> receiver) {
        try {
            String path = baseDir.get() + "/" + year.get() + "/" + month.get() + "/" + day.get() + "/" + hour.get() + "/" + filePattern.get();
            LOG.info("BuildPathFn >> Path: {}", path);
        } catch (Exception e) {
            LOG.error("BuildPathFn >> Exception: {}", e.getMessage()); 

        }
    }
}

运行时输出：

BuildPathFn >> Path: true/true/true/true/true/true

用于构建模板的 mvn 命令的一部分：

 ".... --baseDir --year --month --day --hour --filePattern"

Answer 1

代码很好，是传递参数的正确方式。请记住，该选项需要在模板中定义为 ValueProvider. When staging the template don't pass any parameter that you want to be resolved at runtime. Then the option values should be evaluated when invoking 模板，其中代码已经在描述中（即在请求正文中使用 parameters）。

无法使用 Dataflow REST 传递作业参数 API

Unable to pass job parameters using Dataflow REST API

dataflow

google-cloud-dataflow

apache-beam