Web 抓取使用请求 headers 但它返回页面 html 而不是 Ajax 数据

Question

我正在尝试抓取此网站 https://app.mybodygallery.com/#/

我检查了网络中的 xhr 数据，数据整齐地打包在那里，所以我尝试使用请求 header，但我得到的是页面的 html 而不是数据.

import requests
import json

session = requests.Session()
URL = 'https://app.mybodygallery.com/#/?age=30'

headers ={
    'authority': 'app.mybodygallery.com',
    'method': 'GET',
    'path': '/classes/Photo?where=%7B%22gender%22:%22female%22,%22status%22:%22APPROVED%22,%22weight%22:%7B%22$gte%22:47,%22$lte%22:53%7D%7D',
    'scheme': 'https',
    'accept': 'application/json, text/plain, */*',
    'accept-encoding': 'gzip, deflate, br',
    'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
    'cache-control': 'no-cache',
    'pragma': 'no-cache',
    'referer': URL,
    'sec-ch-ua': '" Not;A Brand";v="99", "Google Chrome";v="91", "Chromium";v="91"',
    'sec-ch-ua-mobile': '?1',
    'sec-fetch-dest': 'empty',
    'sec-fetch-mode': 'cors',
    'sec-fetch-site': 'same-origin',
    'user-agent': 'Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Mobile Safari/537.36'
}

r = session.get(url=URL, headers=headers)
print(r.text)

这就是我的代码，感谢您的帮助

Answer 1

你可以试试那个代码

import requests
import json

# create session
session = requests.Session()

# creat url
url = 'https://app.mybodygallery.com/classes/Photo'

# create params
credentials = {
    'apiKey': 'yFhgrXWYwWoXbH6T6CFhVpROwWaYeldrv7GiDUrZ',
    'apiId': 'LNBLKqkrOMxfDgN18GcdJWfd8scviStNIRpzAoBm'
}

params = {
    'where': json.dumps({"gender": "female", "status": "APPROVED", "age": 30})
}

# change headers
session.headers['x-parse-application-id'] = credentials['apiId']
session.headers['x-parse-javascript-key'] = credentials['apiKey']

# send request
page = session.get(url, params=params).json()
print(json.dumps(page, indent=4))

{
  "results": [
    {
      "objectId": "65e4a670b0",
      "createdAt": "2010-08-15T23:28:56.000Z",
      "updatedAt": "2019-07-23T11:44:11.884Z",
      "user": {
        "__type": "Pointer",
        "className": "_User",
        "objectId": "65e4a7510f"
      },
      "sourceId": 1327,
      "sourceName": "0/863-1",
      "source": "v1Portal",
      "sourceViews": 14852,
      "height": 170,
      "weight": 52.2,
      "age": 30,
      "pant": 30,
      "shirt": 36,
      "bodytype": "hourglass",
      "status": "APPROVED",
      "gender": "female",
      "featured": false,
      "file": {
        "__type": "File",
        "name": "c6767c0e1bb46d997bd0f7b2fcc44f45_logo.jpeg",
        "url": "https://parsefiles.back4app.com/LNBLKqkrOMxfDgN18GcdJWfd8scviStNIRpzAoBm/c6767c0e1bb46d997bd0f7b2fcc44f45_logo.jpeg"
      }
    },
...

P.S。如果我能帮到你 - 请将答案标记为正确 :)

Web 抓取使用请求 headers 但它返回页面 html 而不是 Ajax 数据

Web scraping Using request headers but it's returning page html instead of the Ajax data

python

web-scraping

request

screen-scraping