无法使用线程以正确的方式执行我的脚本

Question

我尝试结合使用 python 和 Thread 创建一个抓取工具来制作执行时间更快。爬虫应该解析所有商店名称及其遍历多个页面的 phone 号码。

脚本运行没有任何问题。由于我是使用 Thread 的新手，所以我很难理解我的做法是否正确。

这是我迄今为止尝试过的方法：

import requests 
from lxml import html
import threading
from urllib.parse import urljoin 

link = "https://www.yellowpages.com/search?search_terms=coffee&geo_location_terms=Los%20Angeles%2C%20CA&page={}"

def get_information(url):
    for pagelink in [url.format(page) for page in range(20)]:
        response = requests.get(pagelink).text
        tree = html.fromstring(response)
        for title in tree.cssselect("div.info"):
            name = title.cssselect("a.business-name span[itemprop=name]")[0].text
            try:
                phone = title.cssselect("div[itemprop=telephone]")[0].text
            except Exception: phone = ""
            print(f'{name} {phone}')

thread = threading.Thread(target=get_information, args=(link,))

thread.start()
thread.join()

The problem being I can't find any difference in time or performance whether I run the above script using Thread or without using Thread. If I'm going wrong, how can I execute the above script using Thread?

编辑：我试图更改逻辑以使用多个链接。现在可以吗？提前致谢。

Answer 1

您可以使用 Threading 并行抓取多个页面，如下所示：

import requests
from lxml import html
import threading
from urllib.parse import urljoin

link = "https://www.yellowpages.com/search?search_terms=coffee&geo_location_terms=Los%20Angeles%2C%20CA&page={}"

def get_information(url):
    response = requests.get(url).text
    tree = html.fromstring(response)
    for title in tree.cssselect("div.info"):
        name = title.cssselect("a.business-name span[itemprop=name]")[0].text
        try:
            phone = title.cssselect("div[itemprop=telephone]")[0].text
        except Exception: phone = ""
        print(f'{name} {phone}')

threads = []
for url in [link.format(page) for page in range(20)]:
    thread = threading.Thread(target=get_information, args=(url,))
    threads.append(thread)
    thread.start()
for thread in threads:
    thread.join()

请注意，不会保留数据序列。这意味着如果逐个抓取页面，提取的数据序列将是：

page_1_name_1
page_1_name_2
page_1_name_3
page_2_name_1
page_2_name_2
page_2_name_3
page_3_name_1
page_3_name_2
page_3_name_3

而线程数据将混合：

page_1_name_1
page_2_name_1
page_1_name_2
page_2_name_2
page_3_name_1
page_2_name_3
page_1_name_3
page_3_name_2
page_3_name_3

无法使用线程以正确的方式执行我的脚本

Unable to execute my script in the right way using thread

python

lxml

web-scraping

python-multithreading

python-3.x