Post 请求不会 return 适当的数据

Question

我试图从一个名为 https://tippmix.hu 的全国博彩网站上抓取一些数据。我想从此特定页面获取数据：https://www.tippmix.hu/sportfogadas#?sportid=999&countryid=99999999&competitionid=45975&page=1. The data is dynamically loaded so I inspected the page and found a specific POST request which is responsible for loading the data I need in json. This is it.

我打开 python 并使用请求库向此页面发出 post 请求：https://api.tippmix.hu/tippmix/search 具有完全相同的 header 和数据，您可以在我以前的照片。不幸的是，我的代码返回了整个 json 文件，就好像我没有指定任何参数一样。这是我的代码（header_convert 函数将复制的 header 字符串转换为字典：

import requests
from header_convert import header_convert

events_url = "https://api.tippmix.hu/tippmix/search"
data = {"fieldValue": "",
        "sportId": "999",
        "competitionGroupId": "99999999",
        "competitionId": "45975",
        "type": "0",
        "date": "0001-01-01T00:00:00.000Z",
        "hitsPerPage": "20",
        "page": "1",
        "minOdds": "null",
        "maxOdds": "null"}

raw_headers = """Accept: application/json, text/plain, */*
Content-Type: application/x-www-form-urlencoded
Origin: https://www.tippmix.hu
Content-Length: 182
Accept-Language: en-us
Host: api.tippmix.hu
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Safari/605.1.15
Referer: https://www.tippmix.hu/
Accept-Encoding: gzip, deflate, br
Connection: keep-alive"""
headers = header_convert.header_convert(raw_headers)
print(headers)

page = requests.post(events_url, data=data, headers=headers)
print(page.content)

这是我的 headers:

{'Accept': 'application/json, text/plain, */*', 'Content-Type': 'application/x-www-form-urlencoded', 'Origin': 'https://www.tippmix.hu', 'Content-Length': '182', 'Accept-Language': 'en-us', 'Host': 'api.tippmix.hu', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Safari/605.1.15', 'Referer': 'https://www.tippmix.hu/', 'Accept-Encoding': 'gzip, deflate, br', 'Connection': 'keep-alive'}

不知是否有人可以帮助我。谢谢！

Answer 1

您需要 post 数据作为 json，使用工作代码更新：

import requests
import json

url = 'https://api.tippmix.hu/tippmix/search'

data = {"fieldValue":"",
    "sportId":999,
    "competitionGroupId":99999999,
    "competitionId":45975,
    "type":0,
    "date":"0001-01-01T00:00:00.000Z",
    "hitsPerPage":20,
    "page":1,
    "minOdds":"null",
    "maxOdds":"null"}

headers = {
    'Accept':'application/json, text/plain, */*',
    'Content-Type':'application/x-www-form-urlencoded',
    'Host':'api.tippmix.hu',
    'Origin':'https://www.tippmix.hu',
    'Referer':'https://www.tippmix.hu/',
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36'
    }

resp = requests.post(url,headers=headers,data=json.dumps(data)).json()
print(resp)

Post 请求不会 return 适当的数据

Post request won't return appropriate data

python

post

web-scraping

python-requests