甚至在使用 selenium python 抓取数据之前网页就已关闭
Web page is closed even before scraping data using selenium python
我正在尝试从 here 抓取新闻头条。该程序开始执行，但是没有提取新闻标题，程序就结束了。
这是我从网站上抓取新闻网址标题的功能
def dvm():
print('--dvm news 360--')
url = 'https://www.dvm360.com/news'
browser.get(url)
time.sleep(20)
headlines = browser.find_elements_by_class_name('title')
url_headlines = [ele.find_element_by_tag_name('a').get_attribute('href') for ele in headlines]
print(url_headlines)
browser.close()
browser.quit()
这是它在我的终端上打印的内容
--dvm news 360--
[15036:4376:0930/121207.716:ERROR:chrome_browser_main_extra_parts_metrics.cc(228)] crbug.com/1216328: Checking Bluetooth availability started. Please report if there is no report that thi
s ends.
[15036:7848:0930/121207.719:ERROR:device_event_log_impl.cc(214)] [12:12:07.719] USB: usb_device_handle_win.cc:1048 Failed to read descriptor from node connection: A device attached to the
system is not functioning. (0x1F)
[15036:4376:0930/121207.720:ERROR:chrome_browser_main_extra_parts_metrics.cc(231)] crbug.com/1216328: Checking Bluetooth availability ended.
[15036:4376:0930/121208.338:ERROR:chrome_browser_main_extra_parts_metrics.cc(234)] crbug.com/1216328: Checking default browser status started. Please report if there is no report that thi
s ends.
[15036:4376:0930/121209.747:ERROR:chrome_browser_main_extra_parts_metrics.cc(238)] crbug.com/1216328: Checking default browser status ended.
[]
这里到底出了什么问题？请帮助我理解。谢谢！
编辑：使用@cruisepandey 的回答后得到的输出如下
--dvm news 360--
[13608:1192:0930/141223.693:ERROR:chrome_browser_main_extra_parts_metrics.cc(228)] crbug.com/1216328: Checking Bluetooth availability started. Please report if there is no report that thi
s ends.
[13608:1192:0930/141223.694:ERROR:chrome_browser_main_extra_parts_metrics.cc(231)] crbug.com/1216328: Checking Bluetooth availability ended.
[13608:1192:0930/141223.695:ERROR:chrome_browser_main_extra_parts_metrics.cc(234)] crbug.com/1216328: Checking default browser status started. Please report if there is no report that thi
s ends.
[13608:12140:0930/141223.700:ERROR:device_event_log_impl.cc(214)] [14:12:23.700] USB: usb_device_handle_win.cc:1048 Failed to read descriptor from node connection: A device attached to th
e system is not functioning. (0x1F)
[13608:1192:0930/141223.827:ERROR:chrome_browser_main_extra_parts_metrics.cc(238)] crbug.com/1216328: Checking default browser status ended.
Morris Animal Foundation announces new equine and animal welfare advisory members
Understanding Cushing syndrome and cortisol: From lab work to rechecks
A call for change: Addressing the lack of diversity in veterinary schools and beyond
Butterfly Network and AVG collaborate to provide breakthrough ultrasound to UrgentVet clinics
3 Methods for battling burnout in veterinary medicine
Dechra acquires veterinary marketing and distributions rights for Equine ProVet APC
something went wrong
我尝试使用无限循环和 xpath 索引来抓取网络应用程序。它根本没有关闭浏览器。该网站本质上是动态的，滚动得越多，您获得的内容就越多。
使用了 Xpath //[contains(@class,'title') and not(@style)]
代码：
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
#driver.implicitly_wait(30)
wait = WebDriverWait(driver, 30)
driver.get("https://www.dvm360.com/news")
try:
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[title='Close']"))).click()
except:
pass
j = 1
try:
while True:
element = driver.find_element(By.XPATH, f"(//*[contains(@class,'title') and not(@style)])[{j}]")
driver.execute_script("arguments[0].scrollIntoView(true);", element)
print(element.get_attribute('innerText'))
j = j + 1
except:
print("something went wrong")
pass
进口：
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
输出：
Morris Animal Foundation announces new equine and animal welfare advisory members
Understanding Cushing syndrome and cortisol: From lab work to rechecks
A call for change: Addressing the lack of diversity in veterinary schools and beyond
Butterfly Network and AVG collaborate to provide breakthrough ultrasound to UrgentVet clinics
3 Methods for battling burnout in veterinary medicine
Dechra acquires veterinary marketing and distributions rights for Equine ProVet APC
Product comparison in an unregulated industry: Hemp
This week on dvm360: 3 ways to combat burnout, plus more veterinary news
Vet’s Best Friend emphasizes mental well-being with company-wide yoga series
How the 'oxygen mask rule' can help combat stress
News-wrap up: This week's veterinary news, plus how Walt Disney would run a veterinary clinic
The Dilemma: Promoting a secure, comfortable workplace
Episode 64: How laughter can rekindle your love for veterinary medicine
NAVC to host first inaugural veterinary nurse summit
AAFCO urges additional research on animal food hemp products
IVPA praises Michigan Supreme Court ruling pertaining to licensed veterinarians
The importance of dental radiology
Dermatologic cytology 101: Tips for collecting samples
10 Things to consider when selling a veterinary practice
Lions and tigers at Smithsonian National Zoo test presumptive positive for COVID-19
3 must-reads on canine IBD
Curb canine anxiety with Calmer Canine
AVMA unveils new “Language of Veterinary Care” online tools and breakthrough research
Subtle signs indicative of an orthopedic issue
This week on dvm360: Calmer Canine for anxious dogs detailed, plus more veterinary news
3 Must-reads on veterinary practice management
Vets Pets unveiling brand-new hospital in Wendell, North Carolina
FDA’s CVM determines TriviumVet’s feline HCM program eligible for expanded conditional approval pathway
Exploring the role of omega-3 supplementation in cats and dogs
News-wrap up: This week's veterinary headlines, plus an update on Australian wildlife impacted by 2019-2020 wildfires
Episode 63: A day in the life of a veterinary toxicologic pathologist
Why mindfulness matters
Stepping up to the plate: Australia works to restore wildlife impacted by wildfires
Vet’s Best Friend to host in-person mental health discussion
Canine Osteosarcoma: Teaching an Old Dog New Tricks
Chewy launches brand-new marketplace service for veterinarians
Animal Medical Center appoints inaugural chief veterinary technician
dvm360 product report: Canine mast cell tumor treatment, plus a pet food line and more
Honoring National Hispanic Heritage Month
Work-life balance versus work integration: What's the difference?
AVMA supports legislation surrounding xylitol warning label requirements
Pet TravelPass introduced to streamline international travel documentation process
Anifera granted seed funding from the CIEL for research
Veterinary cannabis for pain management: The endocannabinoidome
How children can help inspire a more diverse future for veterinary medicine
Antinol partners with dog rescue advocate Lee Asher
3 Must-reads on equine medicine
No more 'Woe is me'
Steering towards antimicrobial stewardship
This week on dvm360: An update on veterinary cannabis for pain management, plus more veterinary news
Resolving itchy skin: Here's what you need to know
Separation anxiety in dogs may affect quality of life
Plugging the protein faucet in dogs with PLE
Image quiz: Liver troubles in a Lab
Sinoscopy in the standing horse
News wrap up: This week's veterinary headlines, plus new advancements in canine cancer testing
Episode 62: Investing in your team is investing in your business
Your food allergy questions answered
Second Basepaws Cat Behavior Summit will deliver leading veterinary expertise into the homes of cat parents
When raising prices is detrimental
Anivive Lifesciences strives to develop 'valley fever' prevention vaccine
Trupanion partners with MCVMA to support veterinary professionals
Student debt: What’s the problem?
PEMCO teams up with Pets Best Pet Insurance to offer pet coverage
Advancements in testing for cancer in canines
Life as a Black veterinary student
如您所见，输出较多，我无法真正将所有内容都粘贴到此处。
如果您想对正在抓取的文章设置限制，可以在 if 条件的帮助下完成，请参见下文。
j = 1
try:
while True:
element = driver.find_element(By.XPATH, f"(//*[contains(@class,'title') and not(@style)])[{j}]")
driver.execute_script("arguments[0].scrollIntoView(true);", element)
print(element.get_attribute('innerText'))
if j ==10:
break
j = j + 1
except:
print("something went wrong")
pass
我正在尝试从 here 抓取新闻头条。该程序开始执行，但是没有提取新闻标题，程序就结束了。 这是我从网站上抓取新闻网址标题的功能
def dvm():
print('--dvm news 360--')
url = 'https://www.dvm360.com/news'
browser.get(url)
time.sleep(20)
headlines = browser.find_elements_by_class_name('title')
url_headlines = [ele.find_element_by_tag_name('a').get_attribute('href') for ele in headlines]
print(url_headlines)
browser.close()
browser.quit()
这是它在我的终端上打印的内容
--dvm news 360--
[15036:4376:0930/121207.716:ERROR:chrome_browser_main_extra_parts_metrics.cc(228)] crbug.com/1216328: Checking Bluetooth availability started. Please report if there is no report that thi
s ends.
[15036:7848:0930/121207.719:ERROR:device_event_log_impl.cc(214)] [12:12:07.719] USB: usb_device_handle_win.cc:1048 Failed to read descriptor from node connection: A device attached to the
system is not functioning. (0x1F)
[15036:4376:0930/121207.720:ERROR:chrome_browser_main_extra_parts_metrics.cc(231)] crbug.com/1216328: Checking Bluetooth availability ended.
[15036:4376:0930/121208.338:ERROR:chrome_browser_main_extra_parts_metrics.cc(234)] crbug.com/1216328: Checking default browser status started. Please report if there is no report that thi
s ends.
[15036:4376:0930/121209.747:ERROR:chrome_browser_main_extra_parts_metrics.cc(238)] crbug.com/1216328: Checking default browser status ended.
[]
这里到底出了什么问题？请帮助我理解。谢谢！
编辑：使用@cruisepandey 的回答后得到的输出如下
--dvm news 360--
[13608:1192:0930/141223.693:ERROR:chrome_browser_main_extra_parts_metrics.cc(228)] crbug.com/1216328: Checking Bluetooth availability started. Please report if there is no report that thi
s ends.
[13608:1192:0930/141223.694:ERROR:chrome_browser_main_extra_parts_metrics.cc(231)] crbug.com/1216328: Checking Bluetooth availability ended.
[13608:1192:0930/141223.695:ERROR:chrome_browser_main_extra_parts_metrics.cc(234)] crbug.com/1216328: Checking default browser status started. Please report if there is no report that thi
s ends.
[13608:12140:0930/141223.700:ERROR:device_event_log_impl.cc(214)] [14:12:23.700] USB: usb_device_handle_win.cc:1048 Failed to read descriptor from node connection: A device attached to th
e system is not functioning. (0x1F)
[13608:1192:0930/141223.827:ERROR:chrome_browser_main_extra_parts_metrics.cc(238)] crbug.com/1216328: Checking default browser status ended.
Morris Animal Foundation announces new equine and animal welfare advisory members
Understanding Cushing syndrome and cortisol: From lab work to rechecks
A call for change: Addressing the lack of diversity in veterinary schools and beyond
Butterfly Network and AVG collaborate to provide breakthrough ultrasound to UrgentVet clinics
3 Methods for battling burnout in veterinary medicine
Dechra acquires veterinary marketing and distributions rights for Equine ProVet APC
something went wrong
我尝试使用无限循环和 xpath 索引来抓取网络应用程序。它根本没有关闭浏览器。该网站本质上是动态的，滚动得越多，您获得的内容就越多。
使用了 Xpath //[contains(@class,'title') and not(@style)]
代码：
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
#driver.implicitly_wait(30)
wait = WebDriverWait(driver, 30)
driver.get("https://www.dvm360.com/news")
try:
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[title='Close']"))).click()
except:
pass
j = 1
try:
while True:
element = driver.find_element(By.XPATH, f"(//*[contains(@class,'title') and not(@style)])[{j}]")
driver.execute_script("arguments[0].scrollIntoView(true);", element)
print(element.get_attribute('innerText'))
j = j + 1
except:
print("something went wrong")
pass
进口：
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
输出：
Morris Animal Foundation announces new equine and animal welfare advisory members
Understanding Cushing syndrome and cortisol: From lab work to rechecks
A call for change: Addressing the lack of diversity in veterinary schools and beyond
Butterfly Network and AVG collaborate to provide breakthrough ultrasound to UrgentVet clinics
3 Methods for battling burnout in veterinary medicine
Dechra acquires veterinary marketing and distributions rights for Equine ProVet APC
Product comparison in an unregulated industry: Hemp
This week on dvm360: 3 ways to combat burnout, plus more veterinary news
Vet’s Best Friend emphasizes mental well-being with company-wide yoga series
How the 'oxygen mask rule' can help combat stress
News-wrap up: This week's veterinary news, plus how Walt Disney would run a veterinary clinic
The Dilemma: Promoting a secure, comfortable workplace
Episode 64: How laughter can rekindle your love for veterinary medicine
NAVC to host first inaugural veterinary nurse summit
AAFCO urges additional research on animal food hemp products
IVPA praises Michigan Supreme Court ruling pertaining to licensed veterinarians
The importance of dental radiology
Dermatologic cytology 101: Tips for collecting samples
10 Things to consider when selling a veterinary practice
Lions and tigers at Smithsonian National Zoo test presumptive positive for COVID-19
3 must-reads on canine IBD
Curb canine anxiety with Calmer Canine
AVMA unveils new “Language of Veterinary Care” online tools and breakthrough research
Subtle signs indicative of an orthopedic issue
This week on dvm360: Calmer Canine for anxious dogs detailed, plus more veterinary news
3 Must-reads on veterinary practice management
Vets Pets unveiling brand-new hospital in Wendell, North Carolina
FDA’s CVM determines TriviumVet’s feline HCM program eligible for expanded conditional approval pathway
Exploring the role of omega-3 supplementation in cats and dogs
News-wrap up: This week's veterinary headlines, plus an update on Australian wildlife impacted by 2019-2020 wildfires
Episode 63: A day in the life of a veterinary toxicologic pathologist
Why mindfulness matters
Stepping up to the plate: Australia works to restore wildlife impacted by wildfires
Vet’s Best Friend to host in-person mental health discussion
Canine Osteosarcoma: Teaching an Old Dog New Tricks
Chewy launches brand-new marketplace service for veterinarians
Animal Medical Center appoints inaugural chief veterinary technician
dvm360 product report: Canine mast cell tumor treatment, plus a pet food line and more
Honoring National Hispanic Heritage Month
Work-life balance versus work integration: What's the difference?
AVMA supports legislation surrounding xylitol warning label requirements
Pet TravelPass introduced to streamline international travel documentation process
Anifera granted seed funding from the CIEL for research
Veterinary cannabis for pain management: The endocannabinoidome
How children can help inspire a more diverse future for veterinary medicine
Antinol partners with dog rescue advocate Lee Asher
3 Must-reads on equine medicine
No more 'Woe is me'
Steering towards antimicrobial stewardship
This week on dvm360: An update on veterinary cannabis for pain management, plus more veterinary news
Resolving itchy skin: Here's what you need to know
Separation anxiety in dogs may affect quality of life
Plugging the protein faucet in dogs with PLE
Image quiz: Liver troubles in a Lab
Sinoscopy in the standing horse
News wrap up: This week's veterinary headlines, plus new advancements in canine cancer testing
Episode 62: Investing in your team is investing in your business
Your food allergy questions answered
Second Basepaws Cat Behavior Summit will deliver leading veterinary expertise into the homes of cat parents
When raising prices is detrimental
Anivive Lifesciences strives to develop 'valley fever' prevention vaccine
Trupanion partners with MCVMA to support veterinary professionals
Student debt: What’s the problem?
PEMCO teams up with Pets Best Pet Insurance to offer pet coverage
Advancements in testing for cancer in canines
Life as a Black veterinary student
如您所见，输出较多，我无法真正将所有内容都粘贴到此处。
如果您想对正在抓取的文章设置限制，可以在 if 条件的帮助下完成，请参见下文。
j = 1
try:
while True:
element = driver.find_element(By.XPATH, f"(//*[contains(@class,'title') and not(@style)])[{j}]")
driver.execute_script("arguments[0].scrollIntoView(true);", element)
print(element.get_attribute('innerText'))
if j ==10:
break
j = j + 1
except:
print("something went wrong")
pass