Kusto - 最新版本行的物化视图

Question

我有一个名为 'metadata' 的 Table，它包含由 TestId 分区的 Parameter 和 ParamterValue 列表。每次更改测试时，该测试都将重新引入到 Azure 数据资源管理器中，并且版本为 never。

我的总体目标是：

定义一个函数 (GetTestsFromSearch)，它接受键值对的参数（动态），这让我可以查询所有测试（最新版本）以获得 Key/Value 对的匹配：

( {{"Search param1", "Search value1"},{"Search param2", "Search value2"}}

示例

GetTestsFromSearch({{"ProjectId", "SturnProject"},{"Product Name", "Nacelle "}})

应该return

TestId	Version
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	3
ea5b688c-b61f-4c5b-bb87-af2eac94d454	1

来自下面的示例元数据 table

另一个目标是创建一个只包含每个测试的最新元数据的物化视图（在下面解释 table）

元数据示例 table

TestId	TestName	Parameter	ParameterValue	Version
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	MyTest	ProjectId	SturnProject	1
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	MyTest	Product Category	2MW	1
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	MyTest	Project Start Date	2022-02-03	1
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	MyTest	ProjectId	SturnProject	2
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	MyTest	Product Category	2MW	2
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	MyTest	Project Start Date	2022-02-03	2
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	MyTest	ProjectId	SturnProject	3
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	MyTest	Product Category	2MW	3
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	MyTest	Project Start Date	2022-02-03	3
ea5b688c-b61f-4c5b-bb87-af2eac94d454	MyTest	ProjectId	SturnProject	1
ea5b688c-b61f-4c5b-bb87-af2eac94d454	MyTest	Project State	Open	1
ea5b688c-b61f-4c5b-bb87-af2eac94d454	MyTest	Product Name	Nacelle	1

随着时间的推移，会有数以千计的测试出现在几个不同的版本中，因此我预计，创建一个只维护每个测试的最新版本 - 我尝试将视图创建为：

metadata
| summarize arg_max(Version,*) by TestId

但这只为每个 TestId/Version 提供了一个参数和参数值，而不是测试的整个结果集。

任何人都可以为我指出这个物化视图的正确方向吗？

我已经包含了一个元数据示例 table 作为数据 Table，它可以直接在 kusto 中使用。

元数据Table作为数据Table

datatable (TestId: string, Name: string, Parameter: string, ParameterValue: string, Version: int) [
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Report, DMS number","1234-231",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project name","Thor3",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","GTRS reference","gtrs",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Product Category","2MW",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project number","TE-12321",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DUT responsible person","ANFRB3",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test execution person","ANFRB3",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project Manager","ANFRB3",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVPR, DMS number","1234-1234",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVRE, DMS number","1231-1213",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Start Date","2022-02-23T00:00:00.0000000Z",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Category","Verification safety",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","GTRS reference","gtrs",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project number","TE-12321",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DUT responsible person","ANFRB2",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test execution person","ANFRB2",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project Manager","ANFRB2",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVPR, DMS number","1234-1234",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVRE, DMS number","1231-1213",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Start Date","2022-02-23T00:00:00.0000000Z",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Category","Verification safety",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Product Category","2MW",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project name","Thor3",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Report, DMS number","1234-231",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Category","Verification safety",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project Manager","ANFRB",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","GTRS reference","gtrs",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Product Category","2MW",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project name","Thor3",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project number","TE-12321",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DUT responsible person","ANFRB",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test execution person","ANFRB",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVPR, DMS number","1234-1234",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Report, DMS number","1234-231",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVRE, DMS number","1231-1213",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Start Date","2022-02-23T00:00:00.0000000Z",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","GTRS reference","gtrs232",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Product Category","4MW",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Project name","Myproject",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Project number","43324534",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","DUT responsible person","ANFRB",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test execution person","ANFRB",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Project Manager","ANFRB",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","DVPR, DMS number","435123454",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test Report, DMS number","123123123",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","DVRE, DMS number","12312312312",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test Start Date","2022-03-01T00:00:00.0000000Z",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test Category","Verification functionality",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test facility","CHE",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test rig","rig23",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Sample ID","1",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Link to test data","asdfsafdsdfa",int(1)
]

谢谢

Answer 1

如果大多数测试包含相同的属性（如您的示例），您可以考虑将架构更改为宽架构，其中每个运行 (Version) 个 TestId 和 Name 是单个记录。结果模式类似于以下输出：

**datatable**
| extend pack(Parameter, ParameterValue)
| summarize make_bag(Column1) by TestId, Name, Version
| evaluate bag_unpack(bag_Column1)

然后，您可以使用以下聚合设置物化视图，它将提供您正在寻找的 IIUC：

T | summarize arg_max(Version, *) by TestId, Name

要从示例中的架构切换到建议的架构，您可以将摄取管道更改为以新的架构格式摄取，或者在更新策略功能中使用 update policy for the transformation. If you choose the latter, avoid using bag_unpack 插件。相反，显式投影您需要的列，以避免 non-deterministic 模式。

另一种方法是将所有属性保留在单个动态列中，结果如下：

**datatable**
| extend pack(Parameter, ParameterValue)
| summarize make_bag(Column1) by TestId, Name, Version

并使用与上述相同的物化视图定义。

对于第一个问题 - 使用第二个建议的架构，您可以尝试如下操作：

let GetTestsFromSearch = (Filter:dynamic)
{
    T
    | extend pack(Parameter, ParameterValue)
    | summarize Properties = make_bag(Column1) by TestId, Name, Version  
    | summarize arg_max(Version, *) by TestId, Name
    | extend Filter
    | mv-apply Filter on 
    ( 
        extend key=tostring(bag_keys(Filter)[0]) 
        | extend expected = tostring(Filter[key]), actual = tostring(Properties[key])
        | summarize count(), countif(actual == expected)
        | where count_  == countif_
    )
};
GetTestsFromSearch(dynamic({"Test Category" : "Verification safety", "Project name" : "Thor3"}));

TestId	Name	Version	Properties	count_	countif_
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	ANFRB-FILEVIEW-TEST	3	{ "Test Report, DMS number": "1234-231", "Project name": "Thor3", "GTRS reference": "gtrs", "Product Category": "2MW", "Project number": "TE-12321", "DUT responsible person": "ANFRB3", "Test execution person": "ANFRB3", "Project Manager": "ANFRB3", "DVPR, DMS number": "1234-1234", "DVRE, DMS number": "1231-1213", "Test Start Date": "2022-02-23T00:00:00.0000000Z", "Test Category": "Verification safety" }	2	2

Kusto - 最新版本行的物化视图

Kusto - Materialized View on latest version of rows

kql

azure-data-explorer

kusto-explorer