Nodejs 无法通过请求承诺请求 url

Question

我无法使用以下代码从 OpenTable 网站获取 https。但是，当我尝试从其他站点获取数据时，例如Instagram 使用相同的代码，它可以工作。因为我刚开始学习使用 node js 进行网络抓取，你能告诉我可能是什么问题吗？

const request = require('request-promise');



(async () => {

    const BASE_URL = 'https://www.opentable.co.uk/s?dateTime=2021-05-30T19%3A00%3A00&covers=2&latitude=51.525225&longitude=-0.079615'


    let response = await request(BASE_URL);

    let $ = cheerio.load(response);
    
    console.log(response);

  
  })();

Answer 1

您的代码很可能不起作用，因为 JavaScript 用于在站点上构建 DOM。或者该站点正在使用反抓取保护。无论如何，我建议你像 Puppeteer:

const puppeteer = require("puppeteer-extra");
const StealthPlugin = require("puppeteer-extra-plugin-stealth");

puppeteer.use(StealthPlugin());

async function scrapeOpenTable() {
  const BASE_URL =
    "https://www.opentable.co.uk/s?dateTime=2021-05-30T19%3A00%3A00&covers=2&latitude=51.525225&longitude=-0.079615";

  const browser = await puppeteer.launch({
    headless: false,
    args: ["--no-sandbox", "--disable-setuid-sandbox"],
  });
  const page = await browser.newPage();
  await page.goto(BASE_URL);
  // More commands here...
  /* You may view the docs at:
       https://pptr.dev/
     And more magic at:
       https://www.npmjs.com/package/puppeteer
     Github:
       https://github.com/puppeteer/puppeteer
  */

  // await browser.close();
}

scrapeOpenTable();

Nodejs 无法通过请求承诺请求 url

Nodejs unable to request url with request-promise

node.js

web-scraping

cheerio

request-promise