在 Foundry Code Repos 中构建管道时出错，代码在预览模式下工作但在构建模式下失败

Question

我们在 Foundry Code Repo 转换中不断收到以下错误。它在预览模式下工作，但在构建模式下失败。

No transforms discovered in the pipeline from the requested files. 
Please add the transform to the pipeline definer. 
If using the Build button in Authoring, please ensure you are running the build from the file where the transform is generated. 
Also note generated transforms may not be discovered using the Authoring Build button, and can be triggered
instead through the Dataset Preview app.: {filesWithoutDatasets=[transforms-python/src/name_of_transform_file.py]}

输入数据集是数据连接器 REST 摄取的结果。还有一个列，我将其称为 jsonResponseColumn，其中包含来自所述摄取的实际 json 响应。代码大致是这样的

from pyspark.sql import functions as F
from pyspark.sql.types import StructType, StructField, ArrayType, StringType, DecimalType
from transforms.api import transform_df, Input, Output

@transform_df(
    Output("output_df_location"),
    input_df=Input("input_df_location"),
)
def compute(input_df):

    schema = create_schema()

    parsed_df = input_df
    parsed_df = parsed_df.withColumn('newField1', F.from_json(parsed_df.jsonResponseColumn, schema, {"mode": "FAILFAST"}))
    parsed_df = parsed_df.withColumn('newField2', F.explode(parsed_df.newField1.fieldInJsonResponse))
    parsed_df = parsed_df.withColumn('newField3', parsed_df.newField2.nestedFieldInJsonResponse)

    parsed_df = parsed_df.withColumn(
        'id', ...
    ).withColumn(
        'key', ...
    ).withColumn(
        .....
    )

    return parsed_df.select(
        ...
    )

def create_schema():
    //basically returns a StructType([...]) that matches the json response from the REST ingest

Answer 1

此 x-form 文件的 sub-folder 在 pipeline.py 中丢失。我添加了缺少的 x-form sub-folder 并且它起作用了。为了让其他开发人员不必在每次添加新 sub-folder 时都更新此文件，我将所有子文件夹移至 /transforms 并将 pipeline.py 更改为下面的内容 &它也有效！

from transforms.api import Pipeline 
import repo_name.transforms as transforms 
my_pipeline = Pipeline() 
my_pipeline.discover_transforms(transforms)

在 Foundry Code Repos 中构建管道时出错，代码在预览模式下工作但在构建模式下失败

Error building pipeline in Foundry Code Repos, Code works in Preview mode but fails in Build mode

pyspark

palantir-foundry

foundry-code-repositories