如何使用 python 3.6 从维基百科类别的所有关联页面中抓取和提取所有子类别名称？

Question

我想抓取类别页面 header 下的所有子类别和页面："Category:Computer science"。其 link 如下所示：http://en.wikipedia.org/wiki/Category:Computer_science.

关于上述问题，我从以下 link 中指定的堆栈溢出答案中得到了一个想法。和

但是，答案并没有完全解决问题。它只会擦除 Pages in category "Computer science"。但是，我想提取所有子类别名称及其相关页面。我希望进程应该以深度为 10 的 BFS 方式报告结果。有什么办法可以做到这一点吗？

我从中找到了以下代码：

from pprint import pprint
from urllib.parse import urljoin

from bs4 import BeautifulSoup
import requests


base_url = 'https://en.wikipedia.org/wiki/Category:Computer science'


def get_next_link(soup):
    return soup.find("a", text="next page")

def extract_links(soup):
    return [a['title'] for a in soup.select("#mw-pages li a")]


with requests.Session() as session:
    content = session.get(base_url).content
    soup = BeautifulSoup(content, 'lxml')

    links = extract_links(soup)
    next_link = get_next_link(soup)
    while next_link is not None:  # while there is a Next Page link
        url = urljoin(base_url, next_link['href'])
        content = session.get(url).content
        soup = BeautifulSoup(content, 'lxml')

        links += extract_links(soup)

        next_link = get_next_link(soup)

pprint(links)

Answer 1

要抓取子类别，您将必须使用 selenium to interact with the dropdowns. A simple traversal over the second category of links will yield the pages, however, to find all the subcategories, recursion is needed to properly group the data. The code below utilizes a simple variant of the breadth-first search 来确定何时停止循环在 while 循环的每次迭代中生成的下拉切换对象：

from selenium import webdriver
import time
from bs4 import BeautifulSoup as soup 
def block_data(_d):
  return {_d.find('h3').text:[[i.a.attrs.get('title'), i.a.attrs.get('href')] for i in _d.find('ul').find_all('li')]}

def get_pages(source:str) -> dict:
  return [block_data(i) for i in soup(source, 'html.parser').find('div', {'id':'mw-pages'}).find_all('div', {'class':'mw-category-group'})]

d = webdriver.Chrome('/path/to/chromedriver')
d.get('https://en.wikipedia.org/wiki/Category:Computer_science')
all_pages = get_pages(d.page_source)
_seen_categories = []
def get_categories(source):
  return [[i['href'], i.text] for i in soup(source, 'html.parser').find_all('a', {'class':'CategoryTreeLabel'})]

def total_depth(c):
  return sum(1 if len(b) ==1 and not b[0] else sum([total_depth(i) for i in b]) for a, b in c.items())

def group_categories(source) -> dict:
  return {i.find('div', {'class':'CategoryTreeItem'}).a.text:(lambda x:None if not x else [group_categories(c) for c in x])(i.find_all('div', {'class':'CategoryTreeChildren'})) for i in source.find_all('div', {'class':'CategoryTreeSection'})}

while True:
  full_dict = group_categories(soup(d.page_source, 'html.parser'))
  flag = False
  for i in d.find_elements_by_class_name('CategoryTreeToggle'):
     try:
       if i.get_attribute('data-ct-title') not in _seen_categories:
          i.click()
          flag = True
          time.sleep(1)
     except:
       pass
     else:
       _seen_categories.append(i.get_attribute('data-ct-title'))
  if not flag:
     break

输出：

all_pages:

[{'\xa0': [['Computer science', '/wiki/Computer_science'], ['Glossary of computer science', '/wiki/Glossary_of_computer_science'], ['Outline of computer science', '/wiki/Outline_of_computer_science']]}, 
 {'B': [['Patrick Baudisch', '/wiki/Patrick_Baudisch'], ['Boolean', '/wiki/Boolean'], ['Business software', '/wiki/Business_software']]}, 
 {'C': [['Nigel A. L. Clarke', '/wiki/Nigel_A._L._Clarke'], ['CLEVER score', '/wiki/CLEVER_score'], ['Computational human modeling', '/wiki/Computational_human_modeling'], ['Computational social choice', '/wiki/Computational_social_choice'], ['Computer engineering', '/wiki/Computer_engineering'], ['Critical code studies', '/wiki/Critical_code_studies']]}, 
  {'I': [['Information and computer science', '/wiki/Information_and_computer_science'], ['Instance selection', '/wiki/Instance_selection'], ['Internet Research (journal)', '/wiki/Internet_Research_(journal)']]}, 
  {'J': [['Jaro–Winkler distance', '/wiki/Jaro%E2%80%93Winkler_distance'], ['User:JUehV/sandbox', '/wiki/User:JUehV/sandbox']]}, 
  {'K': [['Krauss matching wildcards algorithm', '/wiki/Krauss_matching_wildcards_algorithm']]}, 
  {'L': [['Lempel-Ziv complexity', '/wiki/Lempel-Ziv_complexity'], ['Literal (computer programming)', '/wiki/Literal_(computer_programming)']]}, 
  {'M': [['Machine learning in bioinformatics', '/wiki/Machine_learning_in_bioinformatics'], ['Matching wildcards', '/wiki/Matching_wildcards'], ['Sidney Michaelson', '/wiki/Sidney_Michaelson']]}, 
  {'N': [['Nuclear computation', '/wiki/Nuclear_computation']]}, {'O': [['OpenCV', '/wiki/OpenCV']]}, 
  {'P': [['Philosophy of computer science', '/wiki/Philosophy_of_computer_science'], ['Prefetching', '/wiki/Prefetching'], ['Programmer', '/wiki/Programmer']]}, 
  {'Q': [['Quaject', '/wiki/Quaject'], ['Quantum image processing', '/wiki/Quantum_image_processing']]}, 
  {'R': [['Reduction Operator', '/wiki/Reduction_Operator']]}, {'S': [['Social cloud computing', '/wiki/Social_cloud_computing'], ['Software', '/wiki/Software'], ['Computer science in sport', '/wiki/Computer_science_in_sport'], ['Supnick matrix', '/wiki/Supnick_matrix'], ['Symbolic execution', '/wiki/Symbolic_execution']]}, 
  {'T': [['Technology transfer in computer science', '/wiki/Technology_transfer_in_computer_science'], ['Trace Cache', '/wiki/Trace_Cache'], ['Transition (computer science)', '/wiki/Transition_(computer_science)']]}, 
  {'V': [['Viola–Jones object detection framework', '/wiki/Viola%E2%80%93Jones_object_detection_framework'], ['Virtual environment', '/wiki/Virtual_environment'], ['Visual computing', '/wiki/Visual_computing']]}, 
  {'W': [['Wiener connector', '/wiki/Wiener_connector']]}, 
  {'Z': [['Wojciech Zaremba', '/wiki/Wojciech_Zaremba']]}, 
  {'Ρ': [['Portal:Computer science', '/wiki/Portal:Computer_science']]}]

full_dict 相当大，由于它的大小，我无法在此处完全 post，但是，下面是遍历结构和 [=56= 的函数的实现] 深度为十的所有元素：

def trim_data(d, depth, count):
   return {a:None if count == depth else [trim_data(i, depth, count+1) for i in b] for a, b in d.items()}

final_subcategories = trim_data(full_dict, 10, 0)

编辑：从树上移除叶子的脚本：

def remove_empty_children(d):
  return {a:None if len(b) == 1 and not b[0] else 
     [remove_empty_children(i) for i in b if i] for a, b in d.items()}

当运行以上：

c = {'Areas of computer science': [{'Algorithms and data structures': [{'Abstract data types': [{'Priority queues': [{'Heaps (data structures)': [{}]}, {}], 'Heaps (data structures)': [{}]}]}]}]}
d = remove_empty_children(c)

输出：

{'Areas of computer science': [{'Algorithms and data structures': [{'Abstract data types': [{'Priority queues': [{'Heaps (data structures)': None}], 'Heaps (data structures)': None}]}]}]}

编辑 2：展平整个结构：

def flatten_groups(d):
   for a, b in d.items():
     yield a
     if b is not None:
        for i in map(flatten_groups, b):
           yield from i


print(list(flatten_groups(remove_empty_children(c))))

输出：

['Areas of computer science', 'Algorithms and data structures', 'Abstract data types', 'Priority queues', 'Heaps (data structures)', 'Heaps (data structures)']

编辑 3：

要访问特定级别的每个子类别的所有页面，可以使用原始 get_pages 函数和稍微不同的 group_categories 方法

def _group_categories(source) -> dict:
  return {i.find('div', {'class':'CategoryTreeItem'}).find('a')['href']:(lambda x:None if not x else [group_categories(c) for c in x])(i.find_all('div', {'class':'CategoryTreeChildren'})) for i in source.find_all('div', {'class':'CategoryTreeSection'})}
from collections import namedtuple
page = namedtuple('page', ['pages', 'children'])
def subcategory_pages(d, depth, current = 0):
  r = {} 
  for a, b in d.items():
     all_pages_listing = get_pages(requests.get(f'https://en.wikipedia.org{a}').text)
     print(f'page number for {a}: {len(all_pages_listing)}')
     r[a] = page(all_pages_listing, None if current==depth else [subcategory_pages(i, depth, current+1) for i in b])
  return r


print(subcategory_pages(full_dict, 2))

请注意，为了使用 subcategory_pages，必须使用 _group_categories 代替 group_categories。

如何使用 python 3.6 从维基百科类别的所有关联页面中抓取和提取所有子类别名称？

How to scrape and extract all the subcategories names from all its associated pages for a wikipedia category using python 3.6?

python

wikipedia

beautifulsoup

web-scraping

python-3.x