Python文件抛出json解码错误，如何调整

Question

这段代码一直报错，我无法让它保存到文本文件。它一直卡在

回溯（最后一次调用）：文件“c:\Python39\scrape2.py”，第 32 行，位于 response = requests.get(url % page, headers=headers).json() 第 918 行，在 json
提出 RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError：[Errno 期望值]：0

import requests
import json
page = 1
url = f"https://api-prod.grip.events/1/container/4368/search?search=&sort=name&order=asc&type_id=4907,4906,5265,4964,4904,1026,4908&page=%d"
headers = {
    'authority': 'api-prod.grip.events',
    'accept': 'application/json',
    'accept-language': 'en-gb',
    'content-type': 'application/json',
    'if-none-match': 'W/"7132-A/vrxQVW3GqTDiJFLQqx9lN+Y0s"',
    'login-source': 'web',
    'origin': 'https://connect.money2020.com',
    'referer': 'https://connect.money2020.com/money2020europe/app/home/network/list/34589',
    'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="101", "Google Chrome";v="101"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"Windows"',
    'sec-fetch-dest': 'empty',
    'sec-fetch-mode': 'cors',
    'sec-fetch-site': 'cross-site',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36',
    'x-authorization': 'a422cc2a-31fb-4b4e-a1bd-a34b561adc6c',
    'x-grip-version': 'Web/8.3.11',
}


s = requests.Session()

response = requests.post(url, headers=headers)

with open("list.txt", "w") as f:
    for page in range(1, 1000):
        response = requests.get(url % page, headers=headers).json()
        contacts = response("data")
        for contact in contacts:
            target = "%s\t%s\t%s\t%s" % (contact["company_name"], contact["job_title"], contact["name"],  contact["job_industry"])
            f.write(target + "\n")
            print(target)

Answer 1

您确定回复有效吗json？它可能会收到一个错误，而您还没有处理这种情况。

尝试将其更新为以下内容，它应该会打印出任何错误。

with open("list.txt", "w") as f:
    for page in range(1, 1000):
        try:
            response = requests.get(url % page, headers=headers)
            if response.status_code == 200:
                response = response.json()
                contacts = response("data")
                for contact in contacts:
                    target = "%s\t%s\t%s\t%s" % (contact["company_name"], contact["job_title"], contact["name"],  contact["job_industry"])
                    f.write(target + "\n")
                    print(target)
            else:
                print(f"Unsuccessful request: {response}")
        except Exception as e:
            print(f"Error: {e}")

Answer 2

服务器正在返回 HTTP 代码 304（未修改），因为 if-not-match header 已经与 ETag server-side 匹配（因为这可能是从浏览器开发者工具中复制的）。

只需删除此 header（以及其他一些不必要的），并更正拼写错误 (contacts = response["data"])。

import requests
import json

url = "https://api-prod.grip.events/1/container/4368/search?search=&sort=name&order=asc&type_id=4907,4906,5265,4964,4904,1026,4908&page=%d"
headers = {
    'x-authorization': 'a422cc2a-31fb-4b4e-a1bd-a34b561adc6c'
}
with open("list.txt", "w") as f:
    for page in range(1, 1000):
        response = requests.get(url % page, headers=headers).json()
        contacts = response["data"]
        for contact in contacts:
            target = "%s\t%s\t%s\t%s" % (contact["company_name"], contact["job_title"], contact["name"],  contact["job_industry"])
            f.write(target + "\n")
            print(target)

您可能还想查看用于编写 TSV 文件的 csv 模块。

Python文件抛出json解码错误，如何调整

Python file throwing json decode error, how to adjust

python

web-scraping