如何从结果中删除元素标签，网络抓取文章 Python
How to remove element tags from results, Web Scraping Articles with Python
我最近一直在自学 python，而不是 div 直接进入课程，我决定想一些我可以自己研究和工作的剧本想法。在看到视频中提到的类似内容后，我决定做的第一个是网络抓取工具，用于从纽约时报等网站抓取文章。 （我想在 post 之前声明我知道一些网站可能对此有不同的服务条款，我想说清楚我这样做只是为了学习代码的各个方面，没有任何其他动机 -- 我也 有 纽约时报的帐户，并且没有在我没有帐户的网站上这样做）
我对执行此操作所需的 python 有了一些了解，并开始使用一些 BeautifulSoup 命令，其中一些效果很好！我在 F12 检查中找到了引用文章部分的特定元素，并且能够成功地从这些部分中获取文本。
然而，当涉及到文章的正文时，元素的设置方式让我难以抓取所有文本并且没有随附一些标签。
我目前所在的位置：
from bs4 import BeautifulSoup
import requests
source = requests.get('https://www.nytimes.com/2022/01/08/us/teachers-unions-covid-schools.html').text
soup = BeautifulSoup(source, 'lxml')
print('----------')
print(f'TITLE: {soup.title.string}')
print('----------')
print(f'H1: {soup.h1.string}')
print(f'H2: {soup.h2.string}')
print(f'H3: {soup.h3.string}')
print('----------')
article_summary = soup.find('p', class_='css-w6ymp8 e1wiw3jv0').text
print(f'Article summary: {article_summary}')
print('----------')
image_summary = soup.find('span', class_='css-16f3y1r e13ogyst0').text
print(f'Image summary: {image_summary}')
print('----------')
authors = soup.find('p', class_= 'css-aknsld e1jsehar1')
author1 = authors.find('span', class_= 'css-1baulvz').text
author2 = authors.find('span', class_= 'css-1baulvz last-byline').text
print(f'Authors: {author1} and {author2}')
print('----------')
for item in soup.select('.StoryBodyCompanionColumn'):
try:
para = item.find_all('p')
print(para)
except Exception as e:
print('f')
我从中得到的输出是：
----------
TITLE: Teachers’ Unions Push for Remote Schooling, Worrying Democrats - The New York Times
----------
H1: As More Teachers’ Unions Push for Remote Schooling, Parents Worry. So Do Democrats.
H2: The Coronavirus Pandemic: Latest Updates
H3: None
----------
Article summary: Chicago teachers have voted to go remote. Other unions are agitating for change. For Democrats, who promised to keep schools open, the tensions are a distinctly unwelcome development.
----------
Image summary: Alex Brandenburg, an elementary school teacher, protested outside of the Oakland Unified School District headquarters on Friday as part of a sick out.
----------
Authors: Dana Goldstein and Noam Scheiber
----------
[<p class="css-axufdj evys1bk0">Few American cities have labor politics as fraught as Chicago’s, where the nation’s third-largest school system shut down this week after teachers’ union members refused to work in person, arguing that classrooms were unsafe amid the Omicron surge.</p>, <p class="css-axufdj evys1bk0">But in a number of other places, the tenuous labor peace that has allowed most schools to operate normally this year is in danger of collapsing.</p>, <p class="css-axufdj evys1bk0">While not yet threatening to walk off the job, unions are back at negotiating tables, pushing in some cases for a return to remote learning. They frequently cite understaffing because of illness, and shortages of rapid tests and medical-grade masks. Some teachers, in a rear-guard action, have staged sick outs.</p>, <p class="css-axufdj evys1bk0">In Milwaukee, schools are remote until Jan. 18, because of staffing issues. But the teachers’ union president, Amy Mizialko, doubts that the situation will significantly improve<span class="css-8l6xbc evw5hdy0"> </span>and worries that the school board will resist extending online classes.</p>]
[<p class="css-axufdj evys1bk0">“I anticipate it’ll be a fight,” Ms. Mizialko said.</p>, <p class="css-axufdj evys1bk0">She credited the district for at least delaying in-person schooling to start the year but criticized Democratic officials for placing unrealistic pressure on teachers and schools.</p>, <p class="css-axufdj evys1bk0">“I think that Joe Biden and Miguel Cardona and the newly elected mayor of New York City and Lori Lightfoot — they can all declare that schools will be open,” Ms. Mizialko added, referring to the U.S. education secretary and the mayor of Chicago. “But unless they have hundreds of thousands of people to step in for educators who are sick in this uncontrolled surge, they won’t be.”</p>]
[<p class="css-axufdj evys1bk0">For many parents and teachers, the pandemic has become a slog of anxiety over the risk of infection, child care crises, the tedium of school-through-a-screen and, most of all, chronic instability.</p>, <p class="css-axufdj evys1bk0">And for Democrats, the revival of tensions over remote schooling is a distinctly unwelcome development.</p>]
[<p class="css-axufdj evys1bk0">Because they have close ties to the unions, Democrats are concerned that additional closures like those in Chicago could lead to a possible replay of the party’s recent loss in Virginia’s governor race. <a class="css-1g7m0tk" href="https://dfer.org/press/poll-confirms-education-motivating-issue-for-va-voters-in-2021-election-likely-to-be-major-factor-in-midterms/" rel="noopener noreferrer" target="_blank" title="">Polling</a> showed that school disruptions were an important issue for swing voters who broke Republican — particularly suburban white women.</p>, <p class="css-axufdj evys1bk0">“It’s a big deal in most state polling we do,” said Brian Stryker, a partner at the polling firm ALG Research <a class="css-1g7m0tk" href="https://thirdway.imgix.net/pdfs/override/Qualitative-Research-Findings-%E2%80%93-Virginia-Post-Election-Research.pdf" rel="noopener noreferrer" target="_blank" title="">whose work</a> in Virginia indicated that school closures hurt Democrats.</p>, <p class="css-axufdj evys1bk0">“Anyone who thinks this is a political problem that stops at the Chicago city line is kidding themselves,” added Mr. Stryker, whose firm polled for President Biden’s 2020 campaign. “This is going to resonate all across Illinois, across the country.”</p>, <p class="css-axufdj evys1bk0">More than one million of the country’s 50 million public school students<span class="css-8l6xbc evw5hdy0"> </span>were affected by districtwide shutdowns in the first week of January, many of which were announced abruptly and triggered a wave of frustration among parents.</p>, <p class="css-axufdj evys1bk0">“The kids are not the ones that are seriously ill by and large, but we know kids are the ones suffering from remote learning,” said Dan Kirk, whose son attends Walter Payton College Preparatory High School in Chicago, which was closed amid the district’s standoff this week.</p>, <p class="css-axufdj evys1bk0">Several nonunion charter-school networks and districts temporarily transitioned to remote learning after the holidays. But as has been true throughout the pandemic, most of the temporary districtwide closures — including in Detroit, Cleveland, Milwaukee — are taking place in liberal-leaning areas with powerful unions and a more cautious approach to the coronavirus.</p>]
[<p class="css-axufdj evys1bk0">The unions’ demands echo the ones they have made for nearly two years, despite all that has changed. There are now vaccines and <a class="css-1g7m0tk" href="https://www.cdc.gov/coronavirus/2019-ncov/science/science-briefs/transmission_k_12_schools.html#sars-cov-2" rel="noopener noreferrer" target="_blank" title="">the reassuring knowledge</a> that in-school transmission of the virus has been limited.<span class="css-8l6xbc evw5hdy0"> </span>The Omicron variant, while highly contagious, appears to cause less severe illness than previous iterations of Covid-19. </p>]
[<p class="css-axufdj evys1bk0">Most district leaders and many educators say it is imperative for schools to remain open. They cite a large body of research showing that closures harm children, <a class="css-1g7m0tk" href="https://www.nytimes.com/2021/07/28/us/covid-schools-at-home-learning-study.html" title="">academically</a> and <a class="css-1g7m0tk" href="https://www.nytimes.com/2022/01/04/briefing/american-children-crisis-pandemic.html" title="">emotionally</a>, and widen income and racial disparities. </p>, <p class="css-axufdj evys1bk0">But some local union officials are far warier of packed classrooms. In Newark, schools began 2022 with an unexpected stretch of remote learning, set to end on Jan. 18. John Abeigon, the Newark Teachers Union president, said he was hopeful about the return to buildings but that he remained unsure if every school could operate safely.<span class="css-8l6xbc evw5hdy0"> </span>Student vaccination is far from universal, and most parents have not consented to their children taking regular virus tests.</p>, <p class="css-axufdj evys1bk0">Mr. Abeigon said that if tests remain scarce, he might ask for remote learning at specific schools with low vaccination rates and high case counts. He agreed that online learning was a burden to working parents but argued that educators should not be sacrificed for the good of the economy.</p>, <p class="css-axufdj evys1bk0">“I’d see the entire city of Newark unemployed before I allowed one single teacher’s aide to die needlessly,” he said.</p>, <p class="css-axufdj evys1bk0">In Los Angeles, the district has worked closely with the union to keep classrooms open after one of the longest pandemic shutdowns in the country last school year. The vaccination rate for students 12 and older is about 90 percent, with a student vaccine mandate set to <a class="css-1g7m0tk" href="https://www.nytimes.com/2021/12/18/us/los-angeles-vaccine-mandate-delayed.html" title="">kick in this fall</a>. All students and staff are tested for the virus weekly.</p>]
[<p class="css-axufdj evys1bk0">Still, the president of the local union, Cecily Myart-Cruz, would not rule out pushing for a districtwide return to remote learning in the coming weeks. “You know, I want to be honest — I don’t know,” she said.</p>, <p class="css-axufdj evys1bk0">The tensions are not limited to liberal<span class="css-8l6xbc evw5hdy0"> </span>states. In Kentucky, teachers’ unions and at least <a class="css-1g7m0tk" href="https://www.wdrb.com/in-depth/remote-learning-probable-at-some-point-for-jcps-as-covid-19-cases-surge-pollio-says/article_aae24c48-6e40-11ec-846b-5bdbd3d76870.html" rel="noopener noreferrer" target="_blank" title="">one large school district</a> have said they need the flexibility to go remote amid escalating infection rates.</p>, <p class="css-axufdj evys1bk0">But the Republican-controlled state legislature has granted no more than 10 days for such instruction districtwide, and unions there worry that may be inadequate. Jeni Ward Bolander, a leader of a statewide union, said that teachers may have to walk off the job.</p>, <p class="css-axufdj evys1bk0">“Frustration is building on teachers,” Ms. Ward Bolander said. “I hate to say we’d walk out at that point, but it’s absolutely possible.”</p>, <p class="css-axufdj evys1bk0">National teachers’ unions continue to call for classrooms to remain open, but local affiliates hold the most power in negotiations over whether individual districts will close schools.</p>, <p class="css-axufdj evys1bk0">And over the last decade, some locals, including those in Los Angeles and Chicago, were taken over by activist leaders whose tactics can be more aggressive than those of national leaders like <a class="css-1g7m0tk" href="https://www.nytimes.com/2021/02/08/us/schools-reopening-teachers-unions.html" title="">Randi Weingarten</a> of the American Federation of Teachers and <a class="css-1g7m0tk" href="https://www.nytimes.com/2021/12/12/us/politics/teachers-union-becky-pringle.html" title="">Becky Pringle</a> of the National Education Association, both close allies of President Biden.</p>]
[<p class="css-axufdj evys1bk0">Complicating matters, some local unions face internal pressure from their own members. <a class="css-1g7m0tk" href="https://sanfrancisco.cbslocal.com/2022/01/06/covid-oakland-unified-school-district-warns-potential-teacher-sickout/" rel="noopener noreferrer" target="_blank" title="">In the Bay Area</a>, splinter groups of teachers in both Oakland and San Francisco have planned sick outs, and demanded N95 masks, more virus testing and other safety measures.</p>, <p class="itemClass"><strong>The latest Covid data in the U.S.<!-- --> </strong><span>As the Omicron surge causes case counts to reach record highs and hospitalizations to surpass the height of the Delta wave, here’s <a href="https://www.nytimes.com/interactive/2022/01/07/us/covid-data-explained.html?action=click&pgtype=Article&state=default&module=styln-coronavirus&variant=show&region=MAIN_CONTENT_3&block=storyline_levelup_swipe_recirc">how to think about the data</a> and what it’s beginning to show about <a href="https://www.nytimes.com/interactive/2022/01/09/us/omicron-cities-cases-hospitals.html?action=click&pgtype=Article&state=default&module=styln-coronavirus&variant=show&region=MAIN_CONTENT_3&block=storyline_levelup_swipe_recirc">Omicron’s potential toll across the county</a>.</span></p>, <p class="itemClass"><strong>Around the world.<!-- --> </strong><span>In Europe, <a href="https://www.nytimes.com/live/2022/01/10/world/omicron-covid-testing-vaccines/germany-braces-for-more-protests-as-vaccine-rules-tighten-across-europe?action=click&pgtype=Article&state=default&module=styln-coronavirus&variant=show&region=MAIN_CONTENT_3&block=storyline_levelup_swipe_recirc">Germany is bracing for major protests</a> against restrictions after thousands took to the streets in France and Austria, and a tough new vaccine requirement came into force in Italy. In Uganda, <a href="https://www.nytimes.com/2022/01/10/world/africa/uganda-schools-reopen.html?action=click&pgtype=Article&state=default&module=styln-coronavirus&variant=show&region=MAIN_CONTENT_3&block=storyline_levelup_swipe_recirc">schools reopened</a> after the longest pandemic-prompted shutdown in the world.</span></p>, <p class="itemClass"><strong>Staying safe.<!-- --> </strong><span>Worried about spreading Covid? Keep yourself and others safe by following some basic guidance on <a href="https://www.nytimes.com/article/tests-covid-omicron-pcr-rapid.html?action=click&pgtype=Article&state=default&module=styln-coronavirus&variant=show&region=MAIN_CONTENT_3&block=storyline_levelup_swipe_recirc">when to test</a> and <a href="https://www.nytimes.com/article/at-home-covid-tests-accuracy.html?action=click&pgtype=Article&state=default&module=styln-coronavirus&variant=show&region=MAIN_CONTENT_3&block=storyline_levelup_swipe_recirc">how to use at-home virus tests</a> (if you can find them). Here is what to do <a href="https://www.nytimes.com/article/testing-positive-covid-omicron-variant.html?action=click&pgtype=Article&state=default&module=styln-coronavirus&variant=show&region=MAIN_CONTENT_3&block=storyline_levelup_swipe_recirc">if you test positive for the coronavirus</a>.</span></p>, <p class="css-axufdj evys1bk0">Rori Abernethy, a middle-school teacher in San Francisco, organized a sick out there on Thursday. She said the Chicago action had prompted some teachers to ask, “Why isn’t our union doing this?”</p>, <p class="css-axufdj evys1bk0">In Chicago and San Francisco, working-class parents of color disproportionately send their children to the public schools, and they have often supported strict safety measures during the pandemic, including periods of remote learning. And in New York, the nation’s largest school district, schools are operating in person with increased virus testing, with limited dissent from teachers.</p>, <p class="css-axufdj evys1bk0">But the politics become more complicated in suburbs, where union leaders may find themselves at odds with public officials at pains to<span class="css-8l6xbc evw5hdy0"> </span>preserve in-person schooling.</p>, <p class="css-axufdj evys1bk0">In Fairfax County, Virginia’s largest district, the superintendent has <a class="css-1g7m0tk" href="https://www.fcps.edu/return-school/return-school-safety/navigating-january-2022-covid-surge" rel="noopener noreferrer" target="_blank" title="">a plan</a> for switching individual schools to remote learning in the event of many absent teachers.</p>, <p class="css-axufdj evys1bk0">Kimberly Adams, the president of the<span class="css-8l6xbc evw5hdy0"> </span>local education association, said her union may want stricter measures. And she said that districts should be planning for virus surges by distributing devices for potential short bursts of online schooling. </p>, <p class="css-axufdj evys1bk0">But Dan Helmer, a Democratic state delegate whose swing district includes part of Fairfax County, said there was little support among his constituents for a return to online education.</p>]
[<p class="css-axufdj evys1bk0">Deb Andraca, a Democratic state representative in Wisconsin whose district lies just north of Milwaukee, where schools went remote this past week, said that Republicans have targeted her seat and that she expected schools to be a line of attack.</p>, <p class="css-axufdj evys1bk0">“Everyone I know wants schools to stay open,” she said. “But there’s a lot of talk about how teachers’ unions don’t want schools to stay open.”</p>, <p class="css-axufdj evys1bk0">Jim Hobart, a partner at Public Opinion Strategies, a polling firm that counts several Republican senators and governors as clients, said the school closure issue created two advantages for G.O.P. candidates. It has helped narrow their margins among a demographic they’ve traditionally struggled with — white women between their mid-20s and mid-50s — and it has generally undermined Democrats’ claims to competence.</p>, <p class="css-axufdj evys1bk0">“A lot of people — Biden, Mayor Lightfoot in Chicago — have said schools should be open,” Mr. Hobart said. “If they’re not able to prevent schools from choosing to close, that shows a weakness on their part.”</p>, <p class="css-axufdj evys1bk0">Labor officials say that many of their critics are acting in bad faith, exploiting parents’ pandemic-related frustrations to advance longstanding political goals, like discrediting unions and expanding private-school vouchers.</p>, <p class="css-axufdj evys1bk0">Thus far, neither the critiques nor the broader pandemic challenges appear to have significantly hampered unions’ public standing, even according to <a class="css-1g7m0tk" href="https://www.educationnext.org/hunger-for-stability-quells-appetite-for-change-results-2021-education-next-survey-public-opinion-poll/" rel="noopener noreferrer" target="_blank" title="">polls</a> conducted by<span class="css-8l6xbc evw5hdy0"> </span>researchers skeptical of teachers’ unions.</p>]
[<p class="css-axufdj evys1bk0">And if it turns out that Democratic candidates pay a political price for unions’ assertiveness, local labor officials do not consider it to be among their top concerns.</p>, <p class="css-axufdj evys1bk0">If periods of remote learning this winter hurt the Democratic Party, “that’s a question for the consultants and the brain trusts to figure out,” said Mr. Abeigon, the Newark union president. “But that it’s the right thing to do? There’s no question in my mind.”</p>, <p class="css-pncxxs etfikam0">Holly Secon<!-- --> contributed reporting from San Francisco.</p>]
我觉得离完成它很近了，但是我遇到了障碍，所以我希望有人能告诉我如何稍微清理一下。
如果我要这样做：
for item in soup.select('.StoryBodyCompanionColumn'):
try:
para = item.find('p').text
print(para)
except Exception as e:
print('f')
如果我使用 find('p').text 或 find('p').get_text 而不是 find_all('p')，它将要么给我一个失败并打印出 f 要么它只会给我某个 div. 中的第一段（我可以上传它在我执行其他选项时吐出的结果但是有我可以再添加一些，它会大大延长这个 post 甚至更远）（这篇文章有 10 或 11 个不同的 div 正文，每个都有 2-4 个段落每个，而且它们 ALL 拥有相同的 class_= 标签。这确实是我 运行 遇到问题的地方。我也许可以拼接使用一些漂亮的编码来删除标签，这些代码只是从我的结果中删除不需要的标签，但我宁愿有更流畅的代码实际工作而且我知道我遗漏了一些东西。在我继续学习的过程中，非常感谢任何和所有的帮助！
（对于 post 的长度，或者如果这是一个不充分的问题，我深表歉意——我对整个网站和编程都很了解，所以我正处于不知道自己不知道什么的阶段我还不知道，所以如果有一个 link 回答了我的问题而不是你花时间回答那也太棒了，谢谢！）
Select 段落更具体，同时将 p 添加到您的 css selector，而不是段落，您可以简单地调用 .text 或者如果有什么剥离 -> .text.strip() 或 .get_text(strip=True):
for item in soup.select('.StoryBodyCompanionColumn p'):
try:
para = item.text
print(para)
except Exception as e:
print('f')
只需 select 包含所有 p 标签的部分标签并调用 get_text() 获取所有人类可读文本：
soup.select_one('section[name="articleBody"]').get_text(strip=True)
或将 div 和 join() 文本迭代为单个字符串：
' '.join([item.get_text(strip=True) for item in soup.select('.StoryBodyCompanionColumn p')])
例子
from bs4 import BeautifulSoup
import requests
source = requests.get('https://www.nytimes.com/2022/01/08/us/teachers-unions-covid-schools.html').text
soup = BeautifulSoup(source, 'lxml')
soup.select_one('section[name="articleBody"]').get_text(strip=True)
我最近一直在自学 python，而不是 div 直接进入课程，我决定想一些我可以自己研究和工作的剧本想法。在看到视频中提到的类似内容后，我决定做的第一个是网络抓取工具，用于从纽约时报等网站抓取文章。 （我想在 post 之前声明我知道一些网站可能对此有不同的服务条款，我想说清楚我这样做只是为了学习代码的各个方面，没有任何其他动机 -- 我也 有 纽约时报的帐户，并且没有在我没有帐户的网站上这样做）
我对执行此操作所需的 python 有了一些了解，并开始使用一些 BeautifulSoup 命令，其中一些效果很好！我在 F12 检查中找到了引用文章部分的特定元素，并且能够成功地从这些部分中获取文本。
然而，当涉及到文章的正文时，元素的设置方式让我难以抓取所有文本并且没有随附一些标签。
我目前所在的位置：
from bs4 import BeautifulSoup
import requests
source = requests.get('https://www.nytimes.com/2022/01/08/us/teachers-unions-covid-schools.html').text
soup = BeautifulSoup(source, 'lxml')
print('----------')
print(f'TITLE: {soup.title.string}')
print('----------')
print(f'H1: {soup.h1.string}')
print(f'H2: {soup.h2.string}')
print(f'H3: {soup.h3.string}')
print('----------')
article_summary = soup.find('p', class_='css-w6ymp8 e1wiw3jv0').text
print(f'Article summary: {article_summary}')
print('----------')
image_summary = soup.find('span', class_='css-16f3y1r e13ogyst0').text
print(f'Image summary: {image_summary}')
print('----------')
authors = soup.find('p', class_= 'css-aknsld e1jsehar1')
author1 = authors.find('span', class_= 'css-1baulvz').text
author2 = authors.find('span', class_= 'css-1baulvz last-byline').text
print(f'Authors: {author1} and {author2}')
print('----------')
for item in soup.select('.StoryBodyCompanionColumn'):
try:
para = item.find_all('p')
print(para)
except Exception as e:
print('f')
我从中得到的输出是：
----------
TITLE: Teachers’ Unions Push for Remote Schooling, Worrying Democrats - The New York Times
----------
H1: As More Teachers’ Unions Push for Remote Schooling, Parents Worry. So Do Democrats.
H2: The Coronavirus Pandemic: Latest Updates
H3: None
----------
Article summary: Chicago teachers have voted to go remote. Other unions are agitating for change. For Democrats, who promised to keep schools open, the tensions are a distinctly unwelcome development.
----------
Image summary: Alex Brandenburg, an elementary school teacher, protested outside of the Oakland Unified School District headquarters on Friday as part of a sick out.
----------
Authors: Dana Goldstein and Noam Scheiber
----------
[<p class="css-axufdj evys1bk0">Few American cities have labor politics as fraught as Chicago’s, where the nation’s third-largest school system shut down this week after teachers’ union members refused to work in person, arguing that classrooms were unsafe amid the Omicron surge.</p>, <p class="css-axufdj evys1bk0">But in a number of other places, the tenuous labor peace that has allowed most schools to operate normally this year is in danger of collapsing.</p>, <p class="css-axufdj evys1bk0">While not yet threatening to walk off the job, unions are back at negotiating tables, pushing in some cases for a return to remote learning. They frequently cite understaffing because of illness, and shortages of rapid tests and medical-grade masks. Some teachers, in a rear-guard action, have staged sick outs.</p>, <p class="css-axufdj evys1bk0">In Milwaukee, schools are remote until Jan. 18, because of staffing issues. But the teachers’ union president, Amy Mizialko, doubts that the situation will significantly improve<span class="css-8l6xbc evw5hdy0"> </span>and worries that the school board will resist extending online classes.</p>]
[<p class="css-axufdj evys1bk0">“I anticipate it’ll be a fight,” Ms. Mizialko said.</p>, <p class="css-axufdj evys1bk0">She credited the district for at least delaying in-person schooling to start the year but criticized Democratic officials for placing unrealistic pressure on teachers and schools.</p>, <p class="css-axufdj evys1bk0">“I think that Joe Biden and Miguel Cardona and the newly elected mayor of New York City and Lori Lightfoot — they can all declare that schools will be open,” Ms. Mizialko added, referring to the U.S. education secretary and the mayor of Chicago. “But unless they have hundreds of thousands of people to step in for educators who are sick in this uncontrolled surge, they won’t be.”</p>]
[<p class="css-axufdj evys1bk0">For many parents and teachers, the pandemic has become a slog of anxiety over the risk of infection, child care crises, the tedium of school-through-a-screen and, most of all, chronic instability.</p>, <p class="css-axufdj evys1bk0">And for Democrats, the revival of tensions over remote schooling is a distinctly unwelcome development.</p>]
[<p class="css-axufdj evys1bk0">Because they have close ties to the unions, Democrats are concerned that additional closures like those in Chicago could lead to a possible replay of the party’s recent loss in Virginia’s governor race. <a class="css-1g7m0tk" href="https://dfer.org/press/poll-confirms-education-motivating-issue-for-va-voters-in-2021-election-likely-to-be-major-factor-in-midterms/" rel="noopener noreferrer" target="_blank" title="">Polling</a> showed that school disruptions were an important issue for swing voters who broke Republican — particularly suburban white women.</p>, <p class="css-axufdj evys1bk0">“It’s a big deal in most state polling we do,” said Brian Stryker, a partner at the polling firm ALG Research <a class="css-1g7m0tk" href="https://thirdway.imgix.net/pdfs/override/Qualitative-Research-Findings-%E2%80%93-Virginia-Post-Election-Research.pdf" rel="noopener noreferrer" target="_blank" title="">whose work</a> in Virginia indicated that school closures hurt Democrats.</p>, <p class="css-axufdj evys1bk0">“Anyone who thinks this is a political problem that stops at the Chicago city line is kidding themselves,” added Mr. Stryker, whose firm polled for President Biden’s 2020 campaign. “This is going to resonate all across Illinois, across the country.”</p>, <p class="css-axufdj evys1bk0">More than one million of the country’s 50 million public school students<span class="css-8l6xbc evw5hdy0"> </span>were affected by districtwide shutdowns in the first week of January, many of which were announced abruptly and triggered a wave of frustration among parents.</p>, <p class="css-axufdj evys1bk0">“The kids are not the ones that are seriously ill by and large, but we know kids are the ones suffering from remote learning,” said Dan Kirk, whose son attends Walter Payton College Preparatory High School in Chicago, which was closed amid the district’s standoff this week.</p>, <p class="css-axufdj evys1bk0">Several nonunion charter-school networks and districts temporarily transitioned to remote learning after the holidays. But as has been true throughout the pandemic, most of the temporary districtwide closures — including in Detroit, Cleveland, Milwaukee — are taking place in liberal-leaning areas with powerful unions and a more cautious approach to the coronavirus.</p>]
[<p class="css-axufdj evys1bk0">The unions’ demands echo the ones they have made for nearly two years, despite all that has changed. There are now vaccines and <a class="css-1g7m0tk" href="https://www.cdc.gov/coronavirus/2019-ncov/science/science-briefs/transmission_k_12_schools.html#sars-cov-2" rel="noopener noreferrer" target="_blank" title="">the reassuring knowledge</a> that in-school transmission of the virus has been limited.<span class="css-8l6xbc evw5hdy0"> </span>The Omicron variant, while highly contagious, appears to cause less severe illness than previous iterations of Covid-19. </p>]
[<p class="css-axufdj evys1bk0">Most district leaders and many educators say it is imperative for schools to remain open. They cite a large body of research showing that closures harm children, <a class="css-1g7m0tk" href="https://www.nytimes.com/2021/07/28/us/covid-schools-at-home-learning-study.html" title="">academically</a> and <a class="css-1g7m0tk" href="https://www.nytimes.com/2022/01/04/briefing/american-children-crisis-pandemic.html" title="">emotionally</a>, and widen income and racial disparities. </p>, <p class="css-axufdj evys1bk0">But some local union officials are far warier of packed classrooms. In Newark, schools began 2022 with an unexpected stretch of remote learning, set to end on Jan. 18. John Abeigon, the Newark Teachers Union president, said he was hopeful about the return to buildings but that he remained unsure if every school could operate safely.<span class="css-8l6xbc evw5hdy0"> </span>Student vaccination is far from universal, and most parents have not consented to their children taking regular virus tests.</p>, <p class="css-axufdj evys1bk0">Mr. Abeigon said that if tests remain scarce, he might ask for remote learning at specific schools with low vaccination rates and high case counts. He agreed that online learning was a burden to working parents but argued that educators should not be sacrificed for the good of the economy.</p>, <p class="css-axufdj evys1bk0">“I’d see the entire city of Newark unemployed before I allowed one single teacher’s aide to die needlessly,” he said.</p>, <p class="css-axufdj evys1bk0">In Los Angeles, the district has worked closely with the union to keep classrooms open after one of the longest pandemic shutdowns in the country last school year. The vaccination rate for students 12 and older is about 90 percent, with a student vaccine mandate set to <a class="css-1g7m0tk" href="https://www.nytimes.com/2021/12/18/us/los-angeles-vaccine-mandate-delayed.html" title="">kick in this fall</a>. All students and staff are tested for the virus weekly.</p>]
[<p class="css-axufdj evys1bk0">Still, the president of the local union, Cecily Myart-Cruz, would not rule out pushing for a districtwide return to remote learning in the coming weeks. “You know, I want to be honest — I don’t know,” she said.</p>, <p class="css-axufdj evys1bk0">The tensions are not limited to liberal<span class="css-8l6xbc evw5hdy0"> </span>states. In Kentucky, teachers’ unions and at least <a class="css-1g7m0tk" href="https://www.wdrb.com/in-depth/remote-learning-probable-at-some-point-for-jcps-as-covid-19-cases-surge-pollio-says/article_aae24c48-6e40-11ec-846b-5bdbd3d76870.html" rel="noopener noreferrer" target="_blank" title="">one large school district</a> have said they need the flexibility to go remote amid escalating infection rates.</p>, <p class="css-axufdj evys1bk0">But the Republican-controlled state legislature has granted no more than 10 days for such instruction districtwide, and unions there worry that may be inadequate. Jeni Ward Bolander, a leader of a statewide union, said that teachers may have to walk off the job.</p>, <p class="css-axufdj evys1bk0">“Frustration is building on teachers,” Ms. Ward Bolander said. “I hate to say we’d walk out at that point, but it’s absolutely possible.”</p>, <p class="css-axufdj evys1bk0">National teachers’ unions continue to call for classrooms to remain open, but local affiliates hold the most power in negotiations over whether individual districts will close schools.</p>, <p class="css-axufdj evys1bk0">And over the last decade, some locals, including those in Los Angeles and Chicago, were taken over by activist leaders whose tactics can be more aggressive than those of national leaders like <a class="css-1g7m0tk" href="https://www.nytimes.com/2021/02/08/us/schools-reopening-teachers-unions.html" title="">Randi Weingarten</a> of the American Federation of Teachers and <a class="css-1g7m0tk" href="https://www.nytimes.com/2021/12/12/us/politics/teachers-union-becky-pringle.html" title="">Becky Pringle</a> of the National Education Association, both close allies of President Biden.</p>]
[<p class="css-axufdj evys1bk0">Complicating matters, some local unions face internal pressure from their own members. <a class="css-1g7m0tk" href="https://sanfrancisco.cbslocal.com/2022/01/06/covid-oakland-unified-school-district-warns-potential-teacher-sickout/" rel="noopener noreferrer" target="_blank" title="">In the Bay Area</a>, splinter groups of teachers in both Oakland and San Francisco have planned sick outs, and demanded N95 masks, more virus testing and other safety measures.</p>, <p class="itemClass"><strong>The latest Covid data in the U.S.<!-- --> </strong><span>As the Omicron surge causes case counts to reach record highs and hospitalizations to surpass the height of the Delta wave, here’s <a href="https://www.nytimes.com/interactive/2022/01/07/us/covid-data-explained.html?action=click&pgtype=Article&state=default&module=styln-coronavirus&variant=show&region=MAIN_CONTENT_3&block=storyline_levelup_swipe_recirc">how to think about the data</a> and what it’s beginning to show about <a href="https://www.nytimes.com/interactive/2022/01/09/us/omicron-cities-cases-hospitals.html?action=click&pgtype=Article&state=default&module=styln-coronavirus&variant=show&region=MAIN_CONTENT_3&block=storyline_levelup_swipe_recirc">Omicron’s potential toll across the county</a>.</span></p>, <p class="itemClass"><strong>Around the world.<!-- --> </strong><span>In Europe, <a href="https://www.nytimes.com/live/2022/01/10/world/omicron-covid-testing-vaccines/germany-braces-for-more-protests-as-vaccine-rules-tighten-across-europe?action=click&pgtype=Article&state=default&module=styln-coronavirus&variant=show&region=MAIN_CONTENT_3&block=storyline_levelup_swipe_recirc">Germany is bracing for major protests</a> against restrictions after thousands took to the streets in France and Austria, and a tough new vaccine requirement came into force in Italy. In Uganda, <a href="https://www.nytimes.com/2022/01/10/world/africa/uganda-schools-reopen.html?action=click&pgtype=Article&state=default&module=styln-coronavirus&variant=show&region=MAIN_CONTENT_3&block=storyline_levelup_swipe_recirc">schools reopened</a> after the longest pandemic-prompted shutdown in the world.</span></p>, <p class="itemClass"><strong>Staying safe.<!-- --> </strong><span>Worried about spreading Covid? Keep yourself and others safe by following some basic guidance on <a href="https://www.nytimes.com/article/tests-covid-omicron-pcr-rapid.html?action=click&pgtype=Article&state=default&module=styln-coronavirus&variant=show&region=MAIN_CONTENT_3&block=storyline_levelup_swipe_recirc">when to test</a> and <a href="https://www.nytimes.com/article/at-home-covid-tests-accuracy.html?action=click&pgtype=Article&state=default&module=styln-coronavirus&variant=show&region=MAIN_CONTENT_3&block=storyline_levelup_swipe_recirc">how to use at-home virus tests</a> (if you can find them). Here is what to do <a href="https://www.nytimes.com/article/testing-positive-covid-omicron-variant.html?action=click&pgtype=Article&state=default&module=styln-coronavirus&variant=show&region=MAIN_CONTENT_3&block=storyline_levelup_swipe_recirc">if you test positive for the coronavirus</a>.</span></p>, <p class="css-axufdj evys1bk0">Rori Abernethy, a middle-school teacher in San Francisco, organized a sick out there on Thursday. She said the Chicago action had prompted some teachers to ask, “Why isn’t our union doing this?”</p>, <p class="css-axufdj evys1bk0">In Chicago and San Francisco, working-class parents of color disproportionately send their children to the public schools, and they have often supported strict safety measures during the pandemic, including periods of remote learning. And in New York, the nation’s largest school district, schools are operating in person with increased virus testing, with limited dissent from teachers.</p>, <p class="css-axufdj evys1bk0">But the politics become more complicated in suburbs, where union leaders may find themselves at odds with public officials at pains to<span class="css-8l6xbc evw5hdy0"> </span>preserve in-person schooling.</p>, <p class="css-axufdj evys1bk0">In Fairfax County, Virginia’s largest district, the superintendent has <a class="css-1g7m0tk" href="https://www.fcps.edu/return-school/return-school-safety/navigating-january-2022-covid-surge" rel="noopener noreferrer" target="_blank" title="">a plan</a> for switching individual schools to remote learning in the event of many absent teachers.</p>, <p class="css-axufdj evys1bk0">Kimberly Adams, the president of the<span class="css-8l6xbc evw5hdy0"> </span>local education association, said her union may want stricter measures. And she said that districts should be planning for virus surges by distributing devices for potential short bursts of online schooling. </p>, <p class="css-axufdj evys1bk0">But Dan Helmer, a Democratic state delegate whose swing district includes part of Fairfax County, said there was little support among his constituents for a return to online education.</p>]
[<p class="css-axufdj evys1bk0">Deb Andraca, a Democratic state representative in Wisconsin whose district lies just north of Milwaukee, where schools went remote this past week, said that Republicans have targeted her seat and that she expected schools to be a line of attack.</p>, <p class="css-axufdj evys1bk0">“Everyone I know wants schools to stay open,” she said. “But there’s a lot of talk about how teachers’ unions don’t want schools to stay open.”</p>, <p class="css-axufdj evys1bk0">Jim Hobart, a partner at Public Opinion Strategies, a polling firm that counts several Republican senators and governors as clients, said the school closure issue created two advantages for G.O.P. candidates. It has helped narrow their margins among a demographic they’ve traditionally struggled with — white women between their mid-20s and mid-50s — and it has generally undermined Democrats’ claims to competence.</p>, <p class="css-axufdj evys1bk0">“A lot of people — Biden, Mayor Lightfoot in Chicago — have said schools should be open,” Mr. Hobart said. “If they’re not able to prevent schools from choosing to close, that shows a weakness on their part.”</p>, <p class="css-axufdj evys1bk0">Labor officials say that many of their critics are acting in bad faith, exploiting parents’ pandemic-related frustrations to advance longstanding political goals, like discrediting unions and expanding private-school vouchers.</p>, <p class="css-axufdj evys1bk0">Thus far, neither the critiques nor the broader pandemic challenges appear to have significantly hampered unions’ public standing, even according to <a class="css-1g7m0tk" href="https://www.educationnext.org/hunger-for-stability-quells-appetite-for-change-results-2021-education-next-survey-public-opinion-poll/" rel="noopener noreferrer" target="_blank" title="">polls</a> conducted by<span class="css-8l6xbc evw5hdy0"> </span>researchers skeptical of teachers’ unions.</p>]
[<p class="css-axufdj evys1bk0">And if it turns out that Democratic candidates pay a political price for unions’ assertiveness, local labor officials do not consider it to be among their top concerns.</p>, <p class="css-axufdj evys1bk0">If periods of remote learning this winter hurt the Democratic Party, “that’s a question for the consultants and the brain trusts to figure out,” said Mr. Abeigon, the Newark union president. “But that it’s the right thing to do? There’s no question in my mind.”</p>, <p class="css-pncxxs etfikam0">Holly Secon<!-- --> contributed reporting from San Francisco.</p>]
我觉得离完成它很近了，但是我遇到了障碍，所以我希望有人能告诉我如何稍微清理一下。
如果我要这样做：
for item in soup.select('.StoryBodyCompanionColumn'):
try:
para = item.find('p').text
print(para)
except Exception as e:
print('f')
如果我使用 find('p').text 或 find('p').get_text 而不是 find_all('p')，它将要么给我一个失败并打印出 f 要么它只会给我某个 div. 中的第一段（我可以上传它在我执行其他选项时吐出的结果但是有我可以再添加一些，它会大大延长这个 post 甚至更远）（这篇文章有 10 或 11 个不同的 div 正文，每个都有 2-4 个段落每个，而且它们 ALL 拥有相同的 class_= 标签。这确实是我 运行 遇到问题的地方。我也许可以拼接使用一些漂亮的编码来删除标签，这些代码只是从我的结果中删除不需要的标签，但我宁愿有更流畅的代码实际工作而且我知道我遗漏了一些东西。在我继续学习的过程中，非常感谢任何和所有的帮助！
（对于 post 的长度，或者如果这是一个不充分的问题，我深表歉意——我对整个网站和编程都很了解，所以我正处于不知道自己不知道什么的阶段我还不知道，所以如果有一个 link 回答了我的问题而不是你花时间回答那也太棒了，谢谢！）
Select 段落更具体，同时将 p 添加到您的 css selector，而不是段落，您可以简单地调用 .text 或者如果有什么剥离 -> .text.strip() 或 .get_text(strip=True):
for item in soup.select('.StoryBodyCompanionColumn p'):
try:
para = item.text
print(para)
except Exception as e:
print('f')
只需 select 包含所有 p 标签的部分标签并调用 get_text() 获取所有人类可读文本：
soup.select_one('section[name="articleBody"]').get_text(strip=True)
或将 div 和 join() 文本迭代为单个字符串：
' '.join([item.get_text(strip=True) for item in soup.select('.StoryBodyCompanionColumn p')])
例子
from bs4 import BeautifulSoup
import requests
source = requests.get('https://www.nytimes.com/2022/01/08/us/teachers-unions-covid-schools.html').text
soup = BeautifulSoup(source, 'lxml')
soup.select_one('section[name="articleBody"]').get_text(strip=True)