在 2 个 ul 标签之间抓取数据

Question

嗨，我正在尝试在标签之间进行抓取。下面我附上了我想抓取的一部分源代码。如果你仔细看，有 3 个 ul 标签。第一个 ul 标签有 class = "listGroup"。我正在尝试提取第二个“ul”标签的文本，因为它后面跟着另一个具有 class "listGroup" 的 "ul" 标签。请分享我如何做到这一点。

<ul class="listGroup" id="ul_e6d09fbd-19fe-49ac-9b47-bd857c0d411b"><li class="acces-listitems"><a href="https://order.store.mayoclinic.com/books/gnweb43?utm_source=MC-DotOrg-PS&amp;utm_medium=Link&amp;utm_campaign=FamilyHealth-Book&amp;utm_content=FHB">Book: Mayo Clinic Family Health Book, 5th Edition</a></li><li class="acces-hide-listitems"><a href="https://order.store.mayoclinic.com/hl/hldiged?utm_source=MC-DotOrg-PS&amp;utm_medium=Link&amp;utm_campaign=HealthLetter-Digital&amp;utm_content=HLDE">Newsletter: Mayo Clinic Health Letter — Digital Edition</a></li></ul>
<ul>
<li>Osteoporosis</li>
<li>Kidney stones</li>
<li>Excessive urination</li>
<li>Abdominal pain</li>
<li>Tiring easily or weakness</li>
<li>Depression or forgetfulness</li>
<li>Bone and joint pain</li>
<li>Frequent complaints of illness with no apparent cause</li>
<li>Nausea, vomiting or loss of appetite</li>
</ul>
<ul>
<li>A noncancerous growth (adenoma) on a gland is the most common cause.</li>
<li>Enlargement (hyperplasia) of two or more parathyroid glands accounts for most other cases.</li>
<li>A cancerous tumor is a very rare cause of primary hyperparathyroidism.</li>
</ul>

我附上了到目前为止制作的简短脚本。请帮忙.

import requests
import pandas
from bs4 import BeautifulSoup
for link in ['/diseases-conditions/hyperparathyroidism/symptoms-causes/syc-20356194']:
    page = requests.get(f"https://www.mayoclinic.org{link}")
    soup = BeautifulSoup(page.content, "html.parser")
    for each in soup.find_all("ul"):
        print(each)

Answer 1

您可以使用 CSS select 或 ul.listGroup + ul li -> 这将 select <ul> 标签的所有 <li> 标签放在 <ul> 与 class "listGroup":

txt = '''<ul class="listGroup" id="ul_e6d09fbd-19fe-49ac-9b47-bd857c0d411b"><li class="acces-listitems"><a href="https://order.store.mayoclinic.com/books/gnweb43?utm_source=MC-DotOrg-PS&amp;utm_medium=Link&amp;utm_campaign=FamilyHealth-Book&amp;utm_content=FHB">Book: Mayo Clinic Family Health Book, 5th Edition</a></li><li class="acces-hide-listitems"><a href="https://order.store.mayoclinic.com/hl/hldiged?utm_source=MC-DotOrg-PS&amp;utm_medium=Link&amp;utm_campaign=HealthLetter-Digital&amp;utm_content=HLDE">Newsletter: Mayo Clinic Health Letter — Digital Edition</a></li></ul>

<ul>
<li>Osteoporosis</li>
<li>Kidney stones</li>
<li>Excessive urination</li>
<li>Abdominal pain</li>
<li>Tiring easily or weakness</li>
<li>Depression or forgetfulness</li>
<li>Bone and joint pain</li>
<li>Frequent complaints of illness with no apparent cause</li>
<li>Nausea, vomiting or loss of appetite</li>
</ul>
<ul>
<li>A noncancerous growth (adenoma) on a gland is the most common cause.</li>
<li>Enlargement (hyperplasia) of two or more parathyroid glands accounts for most other cases.</li>
<li>A cancerous tumor is a very rare cause of primary hyperparathyroidism.</li>
</ul>'''

soup = BeautifulSoup(txt, 'html.parser')

for li in soup.select('ul.listGroup + ul li'):
    print(li.text)

打印：

Osteoporosis
Kidney stones
Excessive urination
Abdominal pain
Tiring easily or weakness
Depression or forgetfulness
Bone and joint pain
Frequent complaints of illness with no apparent cause
Nausea, vomiting or loss of appetite

Answer 2

这似乎是 CSS select 的自然用例，即：

ul.listGroup + ul li 将 select 第一个 ul 标签中的所有 li 标签，每个 ul 标签后跟 class listGroup。将 + 替换为 ~ 将改为 select 所有 li 标签中的所有（在本例中为 2）ul 标签，每个标签后跟 class listGroup.

要在您的脚本中实施此答案，请将 find_all 替换为 select 并更新 selector 为相关的 CSS selector.

import requests
import pandas
from bs4 import BeautifulSoup
for link in ['/diseases-conditions/hyperparathyroidism/symptoms-causes/syc-20356194']:
    page = requests.get(f"https://www.mayoclinic.org{link}")
    soup = BeautifulSoup(page.content, "html.parser")
    for each in soup.select("ul.listGroup + ul li"):
        print(each.text)

Answer 3

也许您应该考虑使用正则表达式进行捕获。

在 2 个 ul 标签之间抓取数据

scraping data between 2 ul tags

python

html

web-scraping

html-parsing