NiFi ConvertRecord 将嵌套 json 转换为 csv

Question

我必须将嵌套的 json 转换为 csv：输入Json:

[
  {
    "phone": "657-676-6697",
    "city": "London",
    "rc": [],
    "rcName": "Andy"
  },
  {
    "phone": "",
    "city": "Paris",
    "rc": [
      {
        "email": null,
        "firstName": "Mary",
        "lastName": "Kim",
        "title": "Teacher"
      },
      {
        "email": "anna@gmail.com",
        "firstName": "Anna",
        "lastName": "Frank",
        "title": "Student"
      }
    ],
    "rcName": ""
  },{
    "phone": "",
    "city": "",
    "rc": [ {
        "email": "pan@tt",
        "firstName": "Pan",
        "lastName": "Kurt",
        "title": "Teacher"
      }],
    "rcName": "Pan1"
  },
  {
    "phone": "",
    "city": "Oklahoma",
    "rc": [
      {
        "email": null,
        "firstName": "Pearl",
        "lastName": "Knox",
        "title": "Receptionist"
      },
      {
        "email": "Jim@gmail.com",
        "firstName": "Jim",
        "lastName": "Hopkins",
        "title": "TA"
      }
    ],
    "rcName": ""
  }
]

预期输出 csv：

city,lastName,title
  London,
  Paris,Kim,Teacher
  Paris,Frank,Student
  ,Kurt,Teacher
  Oklahoma,Knox,Receptionist
  Oklahome,Hopkins,TA

使用 ConvertRecord 将 phone、city 和 rcName 转换为 csv 对我来说工作得很好。但它为嵌套 json 中的元素提供空值。 link https://www.titanwolf.org/Network/q/2e265d53-43b9-42a4-94ad-6cbcc44cf395/y 上有建议使用 JoltTransformJSON 将输入 JSON 转换为“行”。所以，我正在使用以下规范，

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "rc": {
        "*": {
          "@(2,city)": "[&1].city",
            "@(2,phone)": "[&1].phone",
          "*": "[&1].&"
        }
       }
      }
    }
  }
]

但是，这个 JoltTransform 给出了数组中的所有值，而 ConvertRecord 无法解析它。颠簸变换输出：

[{
    "city": ["Paris", "", "Oklahoma"],
    "phone": ["", "", ""],
    "email": ["pan@tt", null],
    "firstName": ["Mary", "Pan", "Pearl"],
    "lastName": ["Kim", "Kurt", "Knox"],
    "title": ["Teacher", "Teacher", "Receptionist"]
}, {
    "city": ["Paris", "Oklahoma"],
    "phone": ["", ""],
    "email": ["anna@gmail.com", "Jim@gmail.com"],
    "firstName": ["Anna", "Jim"],
    "lastName": ["Frank", "Hopkins"],
    "title": ["Student", "TA"]
}]

我使用了具有以下配置的 ConvertRecord 处理器：记录阅读器：Json路径阅读器记录器：CSVRecordSetWriter

Answer 1

我使用下面的 JoltTransform 规范解决了我的问题

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "rc": {
          "*": {
            "@(2,city)": "[&3].[&1].city",
            "@(2,phone)": "[&3].[&1].phone",
            "*": "[&3].[&1].&"
          }
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": "[]"
      }
    }
  }
]

NiFi ConvertRecord 将嵌套 json 转换为 csv

NiFi ConvertRecord to transform nested json to csv

json

jolt

apache-nifi