如何使用 jq 从复杂的 JSON 文件中获取简单的 JSON 数据文件

Question

我有一个复杂的 json 文件，嵌套到第 4 层和第 5 层，我正在尝试使用 jq 获得以下结果。任何帮助将不胜感激：

{
  "Name": "unix-global",
  "Title": "AWS cli should be installed",
  "desc": "System Package aws-cli should be installed",
  "result": "passed"
} 
{
  "Name": "unix-global",
  "Title": "AWS cli should be installed",
  "desc": "Service besclient should be installed",
  "result": "failed"
}

这是我通过运行 inspec 配置文件获得的 json 文件。真正的目的是将唯一需要的信息提取到一个简单的 json 中，以便我最终可以更新 AWS Redshift 数据库。

{
  "version": "1.7.1",
  "profiles": [{
    "name": "java",
    "title": "InSpec Java in system",
    "maintainer": "awim",
    "copyright": "awim / mtaqwim",
    "copyright_email": "muhammadtaqwiem@gmail.com",
    "license": "All Rights Reserved",
    "summary": "An InSpec Compliance Profile",
    "version": "0.0.1",
    "supports": [],
    "controls": [{
      "title": "identify java in system",
      "desc": "identify java in PATH system",
      "impact": 0.3,
      "refs": [],
      "tags": {},
      "code": "control 'java-1.0' do\n  impact 0.3\n  title 'identify java in system'\n  desc 'identify java in PATH system'\n\n  describe java_info do\n    it{ should exist }\n    its(:version){ should match '1.7'}\n  end\nend",
      "source_location": {
        "ref": "inspec/java/controls/java_1.0.rb",
        "line": 6
      },
      "id": "java-1.0",
      "results": [{
        "status": "passed",
        "code_desc": "java_info should exist",
        "run_time": 0.000895896,
        "start_time": "2017-01-20 05:04:47 +0000"
      }, {
        "status": "passed",
        "code_desc": "java_info version should match \"1.7\"",
        "run_time": 0.067581113,
        "start_time": "2017-01-20 05:04:47 +0000"
      }]
    }, {
      "title": "run java from specific path",
      "desc": "run java from specific path",
      "impact": 1.0,
      "refs": [],
      "tags": {},
      "code": "control 'java-2.0' do\n  impact 1.0\n  title 'run java from specific path'\n  desc 'run java from specific path'\n\n  describe java_info(java_path) do\n    it{ should exist }\n    its(:version){ should match '1.7'}\n  end\nend",
      "source_location": {
        "ref": "inspec/java/controls/java_2.0.rb",
        "line": 8
      },
      "id": "java-2.0",
      "results": [{
        "status": "skipped",
        "code_desc": "java_info",
        "skip_message": "Can't find file \"/opt/jdk/current\"",
        "resource": "java_info",
        "run_time": 1.6512e-05,
        "start_time": "2017-01-20 05:04:47 +0000"
      }]
    }, {
      "title": "identify java home",
      "desc": "identify java home match to specific path",
      "impact": 0.1,
      "refs": [],
      "tags": {},
      "code": "control 'java-3.0' do\n  impact 0.1\n  title 'identify java home'\n  desc 'identify java home match to specific path'\n\n  describe java_info(java_path) do\n    its(:java_home){ should match java_path}\n  end\nend",
      "source_location": {
        "ref": "inspec/java/controls/java_3.0.rb",
        "line": 8
      },
      "id": "java-3.0",
      "results": [{
        "status": "skipped",
        "code_desc": "java_info",
        "skip_message": "Can't find file \"/opt/jdk/current\"",
        "resource": "java_info",
        "run_time": 6.139e-06,
        "start_time": "2017-01-20 05:04:47 +0000"
      }]
    }],
    "groups": [{
      "title": "which(UNIX)/where(Windows) java installed",
      "controls": ["java-1.0"],
      "id": "controls/java_1.0.rb"
    }, {
      "title": "which(UNIX)/where(Windows) java installed",
      "controls": ["java-2.0"],
      "id": "controls/java_2.0.rb"
    }, {
      "title": "which(UNIX)/where(Windows) java installed",
      "controls": ["java-3.0"],
      "id": "controls/java_3.0.rb"
    }],
    "attributes": []
  }],
  "other_checks": [],
  "statistics": {
    "duration": 0.069669698
  }
}

Answer 1

这里有一个 jq 过滤器可以解决这个问题。请注意，过滤器之间的 "piping" 是 必不可少的 。您必须先展平每个父数组，然后再展平它的子数组，否则您会得到它们的笛卡尔积（非常不好）。

jq '.profiles[] 
     | { Name: .name , Controls: .controls[] } 
     | { Name: .Name, Desc: .Controls.desc , Title: .Controls.title , Results: .Controls.results[] } 
     | { Name: .Name, Desc: .Desc , Title: .Title , StartTime: .Results.start_time , RunTime: .Results.run_time , Result: .Results.status }'

为清楚起见，在代码中添加了换行符

输出：

{
  "Name": "java",
  "Desc": "identify java in PATH system",
  "Title": "identify java in system",
  "StartTime": "2017-01-20 05:04:47 +0000",
  "RunTime": 0.000895896,
  "Result": "passed"
}
…etc

一旦您将其展平到此为止，我会考虑将其另存为 CSV，因为这样可以更简单地加载到 Redshift 中。

 jq '.profiles[] 
     | { Name: .name , Controls: .controls[] } 
     | { Name: .Name, Desc: .Controls.desc , Title: .Controls.title , Results: .Controls.results[] } 
     | [ .Name, .Desc , .Title , .Results.start_time , .Results.run_time , .Results.status ] 
     | @csv '

输出：

"\"java\",\"identify java in PATH system\",\"identify java in system\",\"2017-01-20 05:04:47 +0000\",0.000895896,\"passed\""
…etc

如何使用 jq 从复杂的 JSON 文件中获取简单的 JSON 数据文件

How to get a simple JSON data file from a complex JSON file using jq

amazon-redshift

jq

inspec