将网络 activity 错误消息解析为突触字段

Question

我一直在尝试将 Web activity（POST 方法）中的错误记录到突触 table 中的字段中。问题是，message 键字符串中有一些特殊字符，如：

{
    "value": [
        {
            "id": "",
            "runId": "",
            "debugRunId": ,
            "runGroupId": "",
            "pipelineName": "my_dynamic_pipeline_name",
            "parameters": {
                "region_code": "",
                "data_start_date": "",
                "data_end_date": "",
                "etl_insert_batch_id": "",
                "pipeline_subject_area": "",
                "type_of_request": "",
                "pipeline_name": "",
                "pipeline_requested_by": "",
                "debug": "",
                "cdmloadtype": ""
            },
            "invokedBy": {
                "id": "",
                "name": "",
                "invokedByType": ""
            },
            "runStart": "",
            "runEnd": "",
            "durationInMs": ,
            "status": "",
            "message": "Operation on target my_dynamic_pipeline_name failed: Operation on target my_dynamic_dataflow_name failed: {\"StatusCode\":\"DFExecutorUserError\",\"Message\":\"Job failed due to reason: at Sink 'SinkutilFailedDummy': java.sql.BatchUpdateException: Execution Status - FAILED ;Error number - 15165 ;Pipeline name - my_dynamic_pipeline_name; Stored procedure name - my_stored_proc_name ; Error step - Step 3: Key Hash and Util Type Hash Generation ; Insert batch ID - 1816 ; Error Message - Could not find object 'my object' or you do not have permission.\",\"Details\":\"java.sql.BatchUpdateException: Execution Status - FAILED ;Error number - 15165 ;Pipeline name - my_dynamic_pipeline_name; Stored procedure name - my_stored_proc_name ; Error step - Step 3: Key Hash and Util Type Hash Generation ; Insert batch ID - 1816 ; Error Message - Could not find object 'my_object' or you do not have permission.\n\tat shaded.msdataflow.com.microsoft.sqlserver.jdbc.SQLServerStatement.executeBatch(SQLServerStatement.java:1845)\n\tat com.microsoft.dataflow.transformers.store.JDBCWriter.executeBatchSQLs(JDBCStore.scala:462)\n\tat com.microsoft.dataflow.transformers.store.JDBCWriter.executeSQL(JDBCStore.scala:440)\n\tat com.microsoft.dataflow.transformers.store.JDBCWriter$$anonfun$executeTableOpAndPostSQLAndDDL.apply$mcV$sp(JDBCStore.scala:494)\n\tat com.microsoft.dataflow.transformers.store.JDBCWriter$$anonfun$executeTableOpAndPostSQLAndDDL.apply(JDBCStore.scala:494)\n\tat com.microsoft.dataflow.transformers.store.JDBCWriter$$anonfun$executeTableOpAndPostS\"}",
...
}

所以我可以过滤输出：

@activity('pingPL').output.value[0].message

但是有 {} 和 $ 数据流表达式试图计算的特殊字符。我已经尝试在管道表达式或数据流表达式中使用 replace 或 string 函数但没有成功。

有没有办法将其解析为字符串或获取 Message 键？，例如：

@activity('pingPL').output.value[0].message*.failed*.failed.Message

更新：

这似乎有效：

@json(split(activity('pingPL').output.value[0].message, 'failed: ')[2]).Message

我可以按 failed: 拆分，索引 2 会给我 {...} 中的错误日志。我可以将其解析为 json 并使用 Message 键。它正在工作，但它不是理想的动态解决方案，因为错误消息不会始终具有相同的结构。

Answer 1

找到了使用 substring 和 indexof 提取 {...} 信息的解决方案：

substring(activity('pingPL').output.value[0].message,indexof(activity('pingPL').output.value[0].message,'{'),sub(indexof(activity('pingPL').output.value[0].message,'}'),sub(indexof(activity('pingPL').output.value[0].message,'{'),1)))

获取此字符串作为输出：

{\"StatusCode\":\"DFExecutorUserError\",\"Message\":\"Job failed due to reason: at Sink 'SinkutilFailedDummy': java.sql.BatchUpdateException: Execution Status - FAILED ;Error number - 15165 ;Pipeline name - my_dynamic_pipeline_name; Stored procedure name - my_stored_proc_name ; Error step - Step 3: Key Hash and Util Type Hash Generation ; Insert batch ID - 1816 ; Error Message - Could not find object 'my object' or you do not have permission.\",\"Details\":\"java.sql.BatchUpdateException: Execution Status - FAILED ;Error number - 15165 ;Pipeline name - my_dynamic_pipeline_name; Stored procedure name - my_stored_proc_name ; Error step - Step 3: Key Hash and Util Type Hash Generation ; Insert batch ID - 1816 ; Error Message - Could not find object 'my_object' or you do not have permission.\n\tat shaded.msdataflow.com.microsoft.sqlserver.jdbc.SQLServerStatement.executeBatch(SQLServerStatement.java:1845)\n\tat com.microsoft.dataflow.transformers.store.JDBCWriter.executeBatchSQLs(JDBCStore.scala:462)\n\tat com.microsoft.dataflow.transformers.store.JDBCWriter.executeSQL(JDBCStore.scala:440)\n\tat com.microsoft.dataflow.transformers.store.JDBCWriter$$anonfun$executeTableOpAndPostSQLAndDDL.apply$mcV$sp(JDBCStore.scala:494)\n\tat com.microsoft.dataflow.transformers.store.JDBCWriter$$anonfun$executeTableOpAndPostSQLAndDDL.apply(JDBCStore.scala:494)\n\tat com.microsoft.dataflow.transformers.store.JDBCWriter$$anonfun$executeTableOpAndPostS\"}

然后我用json表达式提取关键信息：

json('extracted string').message

然后使用replace删除单引号'以避免sql错误。

这是我提取错误信息的最后一个表达式：

@replace(json(substring(activity('pingPL').output.value[0].message,indexof(activity('pingPL').output.value[0].message,'{'),sub(indexof(activity('pingPL').output.value[0].message,'}'),sub(indexof(activity('pingPL').output.value[0].message,'{'),1)))).message,'''','-')

将网络 activity 错误消息解析为突触字段

Parse a web activity error message into a synapse field

azure-data-factory-2