我如何使用 scrapy with splash 从 javascript.void(0) 抓取 link？

Question

我想获取本站下一页的url: https://cadres.apec.fr/home/mes-offres/recherche-des-offres-demploi/liste-des-offres-demploi.html?sortsType=SCORE&sortsDirection=DESCENDING&nbParPage=20&page=4&lieux=590711&motsCles=commercial&latitude=48.862903&longitude=2.335955

有办法吗？

我试过一些方法，但没有用。

导入scrapy 来自 scrapy_splash 导入 SplashRequest

import splash


class QuotesSpider(scrapy.Spider):
    name = "Spider"
    start_urls = [
        'https://cadres.apec.fr/home/mes-offres/recherche-des-offres-demploi/liste-des-offres-demploi.html?sortsType=SCORE&sortsDirection=DESCENDING&nbParPage=20&page=1&lieux=590711&motsCles=commercial&latitude=48.862903&longitude=2.335955'
    ]
    splash.private_mode_enabled = False

    def start_requests(self):
        for url in self.start_urls:

            yield SplashRequest(url=url, formdata= {'modelStr': json.dumps({'pageSize': 100})},callback=self.parse,args={'wait': 6})

    def parse(self, response):
        links = response.css('span.ng-scope>a::attr(href)').extract()
        urll = ['https://cadres.apec.fr' + link for link in links]
        urls = urll

        for url in urls:
            yield SplashRequest(url=url, callback=self.parse_details,args={'wait': 8, 'private_mode_enabled': False})

    def parse_details(self, response):
        post = response.css('h1.text-uppercase::text').get()
        salary = response.css('div.col-md-6>p::text')[0].extract()
        name = response.css('p.margin-bottom-0 > strong::text').get()
        reference = response.css('p.margin-bottom-5::text').get()
        capturepost = response.css('div.col-md-6>p::text')[1].extract()
        experience = response.css('div.col-md-6>p::text')[2].extract()
        job_status = response.css('div.col-md-6>p::text')[3].extract()
        profile = response.css('[id="profil-recherche"]>p::text').extract()
        company = response.css('[id="entreprise"]>p::text').extract()
        company_1 = '\n'.join(company)
        description = response.css('[id="descriptif-du-poste"]>p::text').extract()
        des = '\n'.join(description)
        list = {"Name": name, 'Salary': salary, 'Post': post, 'Reference': reference, 'Experience': experience,
                'Job Status': job_status, 'Profile': profile, 'Company': company_1, 'Capture of Post': capturepost,
                 'Description': des}

        yield list

如何获得 javascript.void url？

Answer 1

尝试找出总页数并相应地在 URL 中格式化页码。

URL = https://cadres.apec.fr/home/mes-offres/recherche-des-offres-demploi/liste-des-offres-demploi.html?sortsType=SCORE&sortsDirection=DESCENDING&nbParPage=20&page=1&lieux=590711&motsCles=commercial&latitude=48.862903&longitude=2.335955

用可变页面更改 page=1 并迭代项目总数除以每页 20 个项目（页数）。

我如何使用 scrapy with splash 从 javascript.void(0) 抓取 link？

How can i grab link from javascript.void(0) using scrapy with splash?

scrapy

web-scraping

scrapy-splash