根据第一个 key:value 对删除 JSON 个对象
Dedeplicating JSON objects based on 1st key:value pair
这是输出格式，基于“CVE_data_meta”，我需要对匹配的 ID 进行重复数据删除。
#pull references
for ref in item["cve"]["references"]["reference_data"]:
references = ref["url"]
cleanData.append({"CVE_data_meta": cve_data_meta_id,
"description": description,
"baseScore": baseScore,
"vectorSring": vectorString,
"cweID": cweValue,
"cweID URL": ("https://cwe.mitre.org/data/definitions/"
+ str(cweValue) + ".html"),
"references": references,
"publishedDate": pub_date,
"lastModifiedDate": last_mod_date
})
这是我从 API 的清理响应中提取数据并输出到 JSON 文件的迭代：
# # ==========================================================================================
# # narrow response with additional 'keywords'
# # ==========================================================================================
myResults = open("2-cleanData.json", "r")
scope = json.load(myResults)
output_json=[]
results = []
for k in keywords:
counter = 0
items = [x for x in scope if k in x['description']]
for item in items:
output_json.append(item)
counter += 1
results.append(counter)
with open("3-Final CVEs.json", "w+") as outFile2:
outFile2.write(json.dumps(output_json, indent=2,))
keywords变量可由用户更改；但希望任何人都能够添加关键字而不是在输出文件中获得重复的条目。
完整代码here.
示例输出：（3 个 CVE 条目）
{
"CVE_data_meta": "CVE-2021-0924",
"description": "In xhci_vendor_get_ops of xhci.c, there is a possible out of bounds read due to a missing bounds check. This could lead to local escalation of privilege with no additional execution privileges needed. User interaction is not needed for exploitation.Product: AndroidVersions: Android kernelAndroid ID: A-194461020References: Upstream kernel",
"baseScore": 7.8,
"vectorSring": "CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H",
"cweID": "CWE-125",
"cweID URL": "https://cwe.mitre.org/data/definitions/CWE-125.html",
"references": "https://source.android.com/security/bulletin/2021-11-01",
"publishedDate": "2021-12-15T19:15Z",
"lastModifiedDate": "2021-12-17T18:12Z"
},
{
"CVE_data_meta": "CVE-2021-0981",
"description": "In enqueueNotificationInternal of NotificationManagerService.java, there is a possible way to run a foreground service without showing a notification due to improper input validation. This could lead to local escalation of privilege with no additional execution privileges needed. User interaction is not needed for exploitation.Product: AndroidVersions: Android-12Android ID: A-191981182",
"baseScore": 7.8,
"vectorSring": "CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H",
"cweID": "CWE-269",
"cweID URL": "https://cwe.mitre.org/data/definitions/CWE-269.html",
"references": "https://source.android.com/security/bulletin/pixel/2021-12-01",
"publishedDate": "2021-12-15T19:15Z",
"lastModifiedDate": "2021-12-17T18:09Z"
...several entries later...
"CVE_data_meta": "CVE-2021-0924",
"description": "In xhci_vendor_get_ops of xhci.c, there is a possible out of bounds read due to a missing bounds check. This could lead to local escalation of privilege with no additional execution privileges needed. User interaction is not needed for exploitation.Product: AndroidVersions: Android kernelAndroid ID: A-194461020References: Upstream kernel",
"baseScore": 7.8,
"vectorSring": "CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H",
"cweID": "CWE-125",
"cweID URL": "https://cwe.mitre.org/data/definitions/CWE-125.html",
"references": "https://source.android.com/security/bulletin/2021-11-01",
"publishedDate": "2021-12-15T19:15Z",
"lastModifiedDate": "2021-12-17T18:12Z"
},
现在，只需要git删除重复项...
在检查了你的代码后，我相信你可以做这样的事情来避免
重复字典：
results = []
cve_ids = []
for k in keywords:
counter = 0
items = [x for x in scope if k in x['description']]
for item in items if item['cweID'] not in cwe_ids:
output_json.append(item)
cwe_ids.append(item['cweID'])
counter += 1
您可以通过使用 set 跟踪已经看到的 'CVE_data_meta' 条目并跳过已经看到的条目来轻松删除重复结果，如下所示。 set 会员资格测试非常快，所以这会很快。
使用有限的测试数据进行测试：
myResults = [
{'CVE_data_meta': 'CVE-2021-0924',
'description': 'In xhci_vendor_get_ops of xhci.c, there is a possible out of '
'bounds read due to a missing bounds check. This could lead '
'to local escalation of privilege with no additional '
'execution privileges needed. User interaction is not needed '
'for exploitation.Product: AndroidVersions: Android '
'kernelAndroid ID: A-194461020References: Upstream kernel',
'baseScore': 7.8,
'vectorSring': 'CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H',
'cweID': 'CWE-125',
'cweID URL': 'https://cwe.mitre.org/data/definitions/CWE-125.html',
'references': 'https://source.android.com/security/bulletin/2021-11-01',
'publishedDate': '2021-12-15T19:15Z',
'lastModifiedDate': '2021-12-17T18:12Z'},
{'CVE_data_meta': 'CVE-2021-0981',
'description': 'In enqueueNotificationInternal of '
'NotificationManagerService.java, there is a possible way to '
'run a foreground service without showing a notification due '
'to improper input validation. This could lead to local '
'escalation of privilege with no additional execution '
'privileges needed. User interaction is not needed for '
'exploitation.Product: AndroidVersions: Android-12Android ID: '
'A-191981182',
'baseScore': 7.8,
'vectorSring': 'CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H',
'cweID': 'CWE-269',
'cweID URL': 'https://cwe.mitre.org/data/definitions/CWE-269.html',
'references': 'https://source.android.com/security/bulletin/pixel/2021-12-01',
'publishedDate': '2021-12-15T19:15Z',
'lastModifiedDate': '2021-12-17T18:09Z'},
{'CVE_data_meta': 'CVE-2021-0924',
'description': 'In xhci_vendor_get_ops of xhci.c, there is a possible out of '
'bounds read due to a missing bounds check. This could lead '
'to local escalation of privilege with no additional '
'execution privileges needed. User interaction is not needed '
'for exploitation.Product: AndroidVersions: Android '
'kernelAndroid ID: A-194461020References: Upstream kernel',
'baseScore': 7.8,
'vectorSring': 'CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H',
'cweID': 'CWE-125',
'cweID URL': 'https://cwe.mitre.org/data/definitions/CWE-125.html',
'references': 'https://source.android.com/security/bulletin/2021-11-01',
'publishedDate': '2021-12-15T19:15Z',
'lastModifiedDate': '2021-12-17T18:12Z'}
]
代码：
from pprint import pprint
# Deduplicate results
cleaned = []
seen = set()
for obj in myResults:
key = obj['CVE_data_meta']
if key not in seen:
cleaned.append(obj)
seen.add(key)
pprint(cleaned)
这是输出格式，基于“CVE_data_meta”，我需要对匹配的 ID 进行重复数据删除。
#pull references
for ref in item["cve"]["references"]["reference_data"]:
references = ref["url"]
cleanData.append({"CVE_data_meta": cve_data_meta_id,
"description": description,
"baseScore": baseScore,
"vectorSring": vectorString,
"cweID": cweValue,
"cweID URL": ("https://cwe.mitre.org/data/definitions/"
+ str(cweValue) + ".html"),
"references": references,
"publishedDate": pub_date,
"lastModifiedDate": last_mod_date
})
这是我从 API 的清理响应中提取数据并输出到 JSON 文件的迭代：
# # ==========================================================================================
# # narrow response with additional 'keywords'
# # ==========================================================================================
myResults = open("2-cleanData.json", "r")
scope = json.load(myResults)
output_json=[]
results = []
for k in keywords:
counter = 0
items = [x for x in scope if k in x['description']]
for item in items:
output_json.append(item)
counter += 1
results.append(counter)
with open("3-Final CVEs.json", "w+") as outFile2:
outFile2.write(json.dumps(output_json, indent=2,))
keywords变量可由用户更改；但希望任何人都能够添加关键字而不是在输出文件中获得重复的条目。
完整代码here.
示例输出：（3 个 CVE 条目）
{
"CVE_data_meta": "CVE-2021-0924",
"description": "In xhci_vendor_get_ops of xhci.c, there is a possible out of bounds read due to a missing bounds check. This could lead to local escalation of privilege with no additional execution privileges needed. User interaction is not needed for exploitation.Product: AndroidVersions: Android kernelAndroid ID: A-194461020References: Upstream kernel",
"baseScore": 7.8,
"vectorSring": "CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H",
"cweID": "CWE-125",
"cweID URL": "https://cwe.mitre.org/data/definitions/CWE-125.html",
"references": "https://source.android.com/security/bulletin/2021-11-01",
"publishedDate": "2021-12-15T19:15Z",
"lastModifiedDate": "2021-12-17T18:12Z"
},
{
"CVE_data_meta": "CVE-2021-0981",
"description": "In enqueueNotificationInternal of NotificationManagerService.java, there is a possible way to run a foreground service without showing a notification due to improper input validation. This could lead to local escalation of privilege with no additional execution privileges needed. User interaction is not needed for exploitation.Product: AndroidVersions: Android-12Android ID: A-191981182",
"baseScore": 7.8,
"vectorSring": "CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H",
"cweID": "CWE-269",
"cweID URL": "https://cwe.mitre.org/data/definitions/CWE-269.html",
"references": "https://source.android.com/security/bulletin/pixel/2021-12-01",
"publishedDate": "2021-12-15T19:15Z",
"lastModifiedDate": "2021-12-17T18:09Z"
...several entries later...
"CVE_data_meta": "CVE-2021-0924",
"description": "In xhci_vendor_get_ops of xhci.c, there is a possible out of bounds read due to a missing bounds check. This could lead to local escalation of privilege with no additional execution privileges needed. User interaction is not needed for exploitation.Product: AndroidVersions: Android kernelAndroid ID: A-194461020References: Upstream kernel",
"baseScore": 7.8,
"vectorSring": "CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H",
"cweID": "CWE-125",
"cweID URL": "https://cwe.mitre.org/data/definitions/CWE-125.html",
"references": "https://source.android.com/security/bulletin/2021-11-01",
"publishedDate": "2021-12-15T19:15Z",
"lastModifiedDate": "2021-12-17T18:12Z"
},
现在，只需要git删除重复项...
在检查了你的代码后，我相信你可以做这样的事情来避免 重复字典：
results = []
cve_ids = []
for k in keywords:
counter = 0
items = [x for x in scope if k in x['description']]
for item in items if item['cweID'] not in cwe_ids:
output_json.append(item)
cwe_ids.append(item['cweID'])
counter += 1
您可以通过使用 set 跟踪已经看到的 'CVE_data_meta' 条目并跳过已经看到的条目来轻松删除重复结果，如下所示。 set 会员资格测试非常快，所以这会很快。
使用有限的测试数据进行测试：
myResults = [
{'CVE_data_meta': 'CVE-2021-0924',
'description': 'In xhci_vendor_get_ops of xhci.c, there is a possible out of '
'bounds read due to a missing bounds check. This could lead '
'to local escalation of privilege with no additional '
'execution privileges needed. User interaction is not needed '
'for exploitation.Product: AndroidVersions: Android '
'kernelAndroid ID: A-194461020References: Upstream kernel',
'baseScore': 7.8,
'vectorSring': 'CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H',
'cweID': 'CWE-125',
'cweID URL': 'https://cwe.mitre.org/data/definitions/CWE-125.html',
'references': 'https://source.android.com/security/bulletin/2021-11-01',
'publishedDate': '2021-12-15T19:15Z',
'lastModifiedDate': '2021-12-17T18:12Z'},
{'CVE_data_meta': 'CVE-2021-0981',
'description': 'In enqueueNotificationInternal of '
'NotificationManagerService.java, there is a possible way to '
'run a foreground service without showing a notification due '
'to improper input validation. This could lead to local '
'escalation of privilege with no additional execution '
'privileges needed. User interaction is not needed for '
'exploitation.Product: AndroidVersions: Android-12Android ID: '
'A-191981182',
'baseScore': 7.8,
'vectorSring': 'CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H',
'cweID': 'CWE-269',
'cweID URL': 'https://cwe.mitre.org/data/definitions/CWE-269.html',
'references': 'https://source.android.com/security/bulletin/pixel/2021-12-01',
'publishedDate': '2021-12-15T19:15Z',
'lastModifiedDate': '2021-12-17T18:09Z'},
{'CVE_data_meta': 'CVE-2021-0924',
'description': 'In xhci_vendor_get_ops of xhci.c, there is a possible out of '
'bounds read due to a missing bounds check. This could lead '
'to local escalation of privilege with no additional '
'execution privileges needed. User interaction is not needed '
'for exploitation.Product: AndroidVersions: Android '
'kernelAndroid ID: A-194461020References: Upstream kernel',
'baseScore': 7.8,
'vectorSring': 'CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H',
'cweID': 'CWE-125',
'cweID URL': 'https://cwe.mitre.org/data/definitions/CWE-125.html',
'references': 'https://source.android.com/security/bulletin/2021-11-01',
'publishedDate': '2021-12-15T19:15Z',
'lastModifiedDate': '2021-12-17T18:12Z'}
]
代码：
from pprint import pprint
# Deduplicate results
cleaned = []
seen = set()
for obj in myResults:
key = obj['CVE_data_meta']
if key not in seen:
cleaned.append(obj)
seen.add(key)
pprint(cleaned)