如何使用多处理加速此 python 脚本

Question

我有一个从数据帧获取数据的脚本，使用这些数据向网站发出请求，使用 fuzzywuzzy 模块找到确切的 href，然后运行一个函数来收集赔率。我会用多处理模块加速这个脚本，这可能吗？


                           Date       HomeTeam         AwayTeam
0  Monday 6 December 2021 20:00        Everton          Arsenal
1  Monday 6 December 2021 17:30         Empoli          Udinese
2  Monday 6 December 2021 19:45       Cagliari           Torino
3  Monday 6 December 2021 20:00         Getafe  Athletic Bilbao
4  Monday 6 December 2021 15:00  Real Zaragoza            Eibar
5  Monday 6 December 2021 17:15      Cartagena         Tenerife
6  Monday 6 December 2021 20:00         Girona          Leganes
7  Monday 6 December 2021 19:45          Niort         Toulouse
8  Monday 6 December 2021 19:00      Jong Ajax         FC Emmen
9  Monday 6 December 2021 19:00        Jong AZ        Excelsior

脚本

  df = pd.read_excel(path)

  dates = df.Date
  hometeams = df.HomeTeam
  awayteams = df.AwayTeam

  matches_odds = list()

  for i,(a,b,c) in enumerate(zip(dates, hometeams, awayteams)):
      try:
        r = requests.get(f'https://www.betexplorer.com/results/soccer/?year={a.split(" ")[3]}&month={monthToNum(a.split(" ")[2])}&day={a.split(" ")[1]}')
      except requests.exceptions.ConnectionError:
        sleep(10)
        r = requests.get(f'https://www.betexplorer.com/results/soccer/?year={a.split(" ")[3]}&month={monthToNum(a.split(" ")[2])}&day={a.split(" ")[1]}')
      
      soup = BeautifulSoup(r.text, 'html.parser')
      f = soup.find_all('td', class_="table-main__tt")

      for tag in f: 
          match = fuzz.ratio(f'{b} - {c}', tag.find('a').text)
          hour = a.split(" ")[4]
          if hour.split(':')[0] == '23':
              act_hour = '00' + ':' + hour.split(':')[1]
          else:
              act_hour = str(int(hour.split(':')[0]) + 1) + ':' + hour.split(':')[1]
          if match > 70 and act_hour == tag.find('span').text:
              href_id = tag.find('a')['href']

              table = get_odds(href_id)
              matches_odds.append(table)
          
      print(i, ' of ', len(dates))

PS: monthToNum函数只是将月份名称替换为他的数字

Answer 1

首先，您使用输入 i、a、b 和 c 创建循环体的函数。然后，您创建一个 multiprocessing.Pool 并使用适当的参数（i、a、b、c）将此函数提交到池中。

import multiprocessing

df = pd.read_excel(path)

dates = df.Date
hometeams = df.HomeTeam
awayteams = df.AwayTeam

matches_odds = list()

def fetch(data):
    i, (a, b, c) = data
    try:
        r = requests.get(f'https://www.betexplorer.com/results/soccer/?year={a.split(" ")[3]}&month={monthToNum(a.split(" ")[2])}&day={a.split(" ")[1]}')
    except requests.exceptions.ConnectionError:
        sleep(10)
        r = requests.get(f'https://www.betexplorer.com/results/soccer/?year={a.split(" ")[3]}&month={monthToNum(a.split(" ")[2])}&day={a.split(" ")[1]}')
      
    soup = BeautifulSoup(r.text, 'html.parser')
    f = soup.find_all('td', class_="table-main__tt")

    for tag in f: 
        match = fuzz.ratio(f'{b} - {c}', tag.find('a').text)
        hour = a.split(" ")[4]
        if hour.split(':')[0] == '23':
            act_hour = '00' + ':' + hour.split(':')[1]
        else:
            act_hour = str(int(hour.split(':')[0]) + 1) + ':' + hour.split(':')[1]
        if match > 70 and act_hour == tag.find('span').text:
            href_id = tag.find('a')['href']

            table = get_odds(href_id)
            matches_odds.append(table)
          
    print(i, ' of ', len(dates))

if __name__ == '__main__':  
    num_processes = 20
    with multiprocessing.Pool(num_processes) as pool:
        pool.map(fetch, enumerate(zip(dates, hometeams, awayteams)))

此外，multiprocessing并不是提高速度的唯一方法。异步编程也可以使用，并且可能更适合这种情况，尽管 multiprocessing 也可以完成这项工作 - 只是想提一下。

如果仔细阅读Python multiprocessing documentation，就会一目了然。

如何使用多处理加速此 python 脚本

How to speed up this python script with multiprocessing

python

python-multiprocessing