抓取多个 url 时多次迭代 returns 相同的数据
Multiple iteration returns same data when scraping multiple urls
我有一组 urls 大约 170。所有 urls 来自同一个网站 www.blackoffer.com，它包含文章关于不同的主题。
我的任务是从页面中提取文章标题和段落。
我正在使用 selenium==3.141.0 来抓取数据。当我编写代码和 运行 '.py' 文件时。它 returns 所有 170 个文本文件（我想要的），但它 returns 所有 170 个文件中只有相同的数据。
这是我的代码
# Import required libraries
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
# Set options to display the window of chrome
options = Options()
options.headless = True
options.add_argument('window-size=1920x1080')
# Multiple urls
urls = ['https://insights.blackcoffer.com/how-is-login-logout-time-tracking-for-employees-in-office-done-by-ai/',
'https://insights.blackcoffer.com/how-does-ai-help-to-monitor-retail-shelf-watches/',
'https://insights.blackcoffer.com/ai-and-its-impact-on-the-fashion-industry/',
'https://insights.blackcoffer.com/how-do-deep-learning-models-predict-old-and-new-drugs-that-are-successfully-treated-in-healthcare/',
'https://insights.blackcoffer.com/how-artificial-intelligence-can-boost-your-productivity-level/',
'https://insights.blackcoffer.com/how-are-genetic-sequencing-maps-affected-by-deep-learning-and-ai/',
'https://insights.blackcoffer.com/how-is-ai-used-to-solve-traffic-management/',
'https://insights.blackcoffer.com/benefits-of-big-data-in-different-fields/',
'https://insights.blackcoffer.com/how-big-data-will-impact-the-future-of-business/',
'https://insights.blackcoffer.com/how-will-ai-make-decisions-in-tomorrows-wars/',
'https://insights.blackcoffer.com/which-one-is-better-ai-or-big-data/',
'https://insights.blackcoffer.com/how-robots-can-help-in-e-learning-platforms/',
'https://insights.blackcoffer.com/how-does-big-data-help-in-finance-and-the-growth-of-large-firms/',
'https://insights.blackcoffer.com/future-of-work-robot-ai-and-automation/',
'https://insights.blackcoffer.com/how-ai-will-help-the-defense-power-of-a-country/',
'https://insights.blackcoffer.com/future-of-ai-and-machine-roles-in-the-medical-sector/',
'https://insights.blackcoffer.com/ai-in-healthcare-to-improve-patient-outcomes/',
'https://insights.blackcoffer.com/what-if-the-creation-is-taking-over-the-creator/',
'https://insights.blackcoffer.com/what-jobs-will-robots-take-from-humans-in-the-future/',
'https://insights.blackcoffer.com/will-machine-replace-the-human-in-the-future-of-work/',
'https://insights.blackcoffer.com/will-ai-replace-us-or-work-with-us/',
'https://insights.blackcoffer.com/man-and-machines-together-machines-are-more-diligent-than-humans-blackcoffe/',
'https://insights.blackcoffer.com/in-future-or-in-upcoming-years-humans-and-machines-are-going-to-work-together-in-every-field-of-work/',
'https://insights.blackcoffer.com/how-neural-networks-can-be-applied-in-various-areas-in-the-future/',
'https://insights.blackcoffer.com/how-machine-learning-will-affect-your-business/',
'https://insights.blackcoffer.com/deep-learning-impact-on-areas-of-e-learning/',
'https://insights.blackcoffer.com/how-to-protect-future-data-and-its-privacy-blackcoffer/',
'https://insights.blackcoffer.com/how-machines-ai-automations-and-robo-human-are-effective-in-finance-and-banking/',
'https://insights.blackcoffer.com/ai-human-robotics-machine-future-planet-blackcoffer-thinking-jobs-workplace/',
'https://insights.blackcoffer.com/how-ai-will-change-the-world-blackcoffer/',
'https://insights.blackcoffer.com/future-of-work-how-ai-has-entered-the-workplace/',
'https://insights.blackcoffer.com/ai-tool-alexa-google-assistant-finance-banking-tool-future/',
'https://insights.blackcoffer.com/ai-healthcare-revolution-ml-technology-algorithm-google-analytics-industrialrevolution/',
'https://insights.blackcoffer.com/all-you-need-to-know-about-online-marketing/',
'https://insights.blackcoffer.com/evolution-of-advertising-industry/',
'https://insights.blackcoffer.com/how-data-analytics-can-help-your-business-respond-to-the-impact-of-covid-19/',
'https://insights.blackcoffer.com/covid-19-environmental-impact-for-the-future/',
'https://insights.blackcoffer.com/environmental-impact-of-the-covid-19-pandemic-lesson-for-the-future/',
'https://insights.blackcoffer.com/how-data-analytics-and-ai-are-used-to-halt-the-covid-19-pandemic/',
'https://insights.blackcoffer.com/difference-between-artificial-intelligence-machine-learning-statistics-and-data-mining/',
'https://insights.blackcoffer.com/how-python-became-the-first-choice-for-data-science/',
'https://insights.blackcoffer.com/how-google-fit-measure-heart-and-respiratory-rates-using-a-phone/',
'https://insights.blackcoffer.com/what-is-the-future-of-mobile-apps/',
'https://insights.blackcoffer.com/impact-of-ai-in-health-and-medicine/',
'https://insights.blackcoffer.com/telemedicine-what-patients-like-and-dislike-about-it/',
'https://insights.blackcoffer.com/how-we-forecast-future-technologies/',
'https://insights.blackcoffer.com/can-robots-tackle-late-life-loneliness/',
'https://insights.blackcoffer.com/embedding-care-robots-into-society-socio-technical-considerations/',
'https://insights.blackcoffer.com/management-challenges-for-future-digitalization-of-healthcare-services/',
'https://insights.blackcoffer.com/are-we-any-closer-to-preventing-a-nuclear-holocaust/',
'https://insights.blackcoffer.com/will-technology-eliminate-the-need-for-animal-testing-in-drug-development/',
'https://insights.blackcoffer.com/will-we-ever-understand-the-nature-of-consciousness/',
'https://insights.blackcoffer.com/will-we-ever-colonize-outer-space/',
'https://insights.blackcoffer.com/what-is-the-chance-homo-sapiens-will-survive-for-the-next-500-years/',
'https://insights.blackcoffer.com/why-does-your-business-need-a-chatbot/',
'https://insights.blackcoffer.com/how-you-lead-a-project-or-a-team-without-any-technical-expertise/',
'https://insights.blackcoffer.com/can-you-be-great-leader-without-technical-expertise/',
'https://insights.blackcoffer.com/how-does-artificial-intelligence-affect-the-environment/',
'https://insights.blackcoffer.com/how-to-overcome-your-fear-of-making-mistakes-2/',
'https://insights.blackcoffer.com/is-perfection-the-greatest-enemy-of-productivity/',
'https://insights.blackcoffer.com/global-financial-crisis-2008-causes-effects-and-its-solution/',
'https://insights.blackcoffer.com/gender-diversity-and-equality-in-the-tech-industry/',
'https://insights.blackcoffer.com/how-to-overcome-your-fear-of-making-mistakes/',
'https://insights.blackcoffer.com/how-small-business-can-survive-the-coronavirus-crisis/',
'https://insights.blackcoffer.com/impacts-of-covid-19-on-vegetable-vendors-and-food-stalls/',
'https://insights.blackcoffer.com/impacts-of-covid-19-on-vegetable-vendors/',
'https://insights.blackcoffer.com/impact-of-covid-19-pandemic-on-tourism-aviation-industries/',
'https://insights.blackcoffer.com/impact-of-covid-19-pandemic-on-sports-events-around-the-world/',
'https://insights.blackcoffer.com/changing-landscape-and-emerging-trends-in-the-indian-it-ites-industry/',
'https://insights.blackcoffer.com/online-gaming-adolescent-online-gaming-effects-demotivated-depression-musculoskeletal-and-psychosomatic-symptoms/',
'https://insights.blackcoffer.com/human-rights-outlook/',
'https://insights.blackcoffer.com/how-voice-search-makes-your-business-a-successful-business/',
'https://insights.blackcoffer.com/how-the-covid-19-crisis-is-redefining-jobs-and-services/',
'https://insights.blackcoffer.com/how-to-increase-social-media-engagement-for-marketers/',
'https://insights.blackcoffer.com/impacts-of-covid-19-on-streets-sides-food-stalls/',
'https://insights.blackcoffer.com/coronavirus-impact-on-energy-markets-2/',
'https://insights.blackcoffer.com/coronavirus-impact-on-the-hospitality-industry-5/',
'https://insights.blackcoffer.com/lessons-from-the-past-some-key-learnings-relevant-to-the-coronavirus-crisis-4/',
'https://insights.blackcoffer.com/estimating-the-impact-of-covid-19-on-the-world-of-work-2/',
'https://insights.blackcoffer.com/estimating-the-impact-of-covid-19-on-the-world-of-work-3/',
'https://insights.blackcoffer.com/travel-and-tourism-outlook/',
'https://insights.blackcoffer.com/gaming-disorder-and-effects-of-gaming-on-health/',
'https://insights.blackcoffer.com/what-is-the-repercussion-of-the-environment-due-to-the-covid-19-pandemic-situation/',
'https://insights.blackcoffer.com/what-is-the-repercussion-of-the-environment-due-to-the-covid-19-pandemic-situation-2/',
'https://insights.blackcoffer.com/impact-of-covid-19-pandemic-on-office-space-and-co-working-industries/',
'https://insights.blackcoffer.com/contribution-of-handicrafts-visual-arts-literature-in-the-indian-economy/',
'https://insights.blackcoffer.com/how-covid-19-is-impacting-payment-preferences/',
'https://insights.blackcoffer.com/how-will-covid-19-affect-the-world-of-work-2/',
'https://insights.blackcoffer.com/lessons-from-the-past-some-key-learnings-relevant-to-the-coronavirus-crisis/',
'https://insights.blackcoffer.com/covid-19-how-have-countries-been-responding/',
'https://insights.blackcoffer.com/coronavirus-impact-on-the-hospitality-industry-2/',
'https://insights.blackcoffer.com/how-will-covid-19-affect-the-world-of-work-3/',
'https://insights.blackcoffer.com/coronavirus-impact-on-the-hospitality-industry-3/',
'https://insights.blackcoffer.com/estimating-the-impact-of-covid-19-on-the-world-of-work/',
'https://insights.blackcoffer.com/covid-19-how-have-countries-been-responding-2/',
'https://insights.blackcoffer.com/how-will-covid-19-affect-the-world-of-work-4/',
'https://insights.blackcoffer.com/lessons-from-the-past-some-key-learnings-relevant-to-the-coronavirus-crisis-2/',
'https://insights.blackcoffer.com/lessons-from-the-past-some-key-learnings-relevant-to-the-coronavirus-crisis-3/',
'https://insights.blackcoffer.com/coronavirus-impact-on-the-hospitality-industry-4/',
'https://insights.blackcoffer.com/why-scams-like-nirav-modi-happen-with-indian-banks/',
'https://insights.blackcoffer.com/impact-of-covid-19-on-the-global-economy/',
'https://insights.blackcoffer.com/impact-of-covid-19coronavirus-on-the-indian-economy-2/',
'https://insights.blackcoffer.com/impact-of-covid-19-on-the-global-economy-2/',
'https://insights.blackcoffer.com/impact-of-covid-19-coronavirus-on-the-indian-economy-3/',
'https://insights.blackcoffer.com/should-celebrities-be-allowed-to-join-politics/',
'https://insights.blackcoffer.com/how-prepared-is-india-to-tackle-a-possible-covid-19-outbreak/',
'https://insights.blackcoffer.com/how-will-covid-19-affect-the-world-of-work/',
'https://insights.blackcoffer.com/controversy-as-a-marketing-strategy/',
'https://insights.blackcoffer.com/coronavirus-impact-on-the-hospitality-industry/',
'https://insights.blackcoffer.com/coronavirus-impact-on-energy-markets/',
'https://insights.blackcoffer.com/what-are-the-key-policies-that-will-mitigate-the-impacts-of-covid-19-on-the-world-of-work/',
'https://insights.blackcoffer.com/marketing-drives-results-with-a-focus-on-problems/',
'https://insights.blackcoffer.com/continued-demand-for-sustainability/',
'https://insights.blackcoffer.com/coronavirus-disease-covid-19-effect-the-impact-and-role-of-mass-media-during-the-pandemic/',
'https://insights.blackcoffer.com/should-people-wear-fabric-gloves-seeking-evidence-regarding-the-differential-transfer-of-covid-19-or-coronaviruses-generally-between-surfaces/',
'https://insights.blackcoffer.com/why-is-there-a-severe-immunological-and-inflammatory-explosion-in-those-affected-by-sarms-covid-19/',
'https://insights.blackcoffer.com/what-do-you-think-is-the-lesson-or-lessons-to-be-learned-with-covid-19/',
'https://insights.blackcoffer.com/coronavirus-the-unexpected-challenge-for-the-european-union/',
'https://insights.blackcoffer.com/industrial-revolution-4-0-pros-and-cons/',
'https://insights.blackcoffer.com/impact-of-covid-19-coronavirus-on-the-indian-economy/',
'https://insights.blackcoffer.com/impact-of-covid-19-coronavirus-on-the-indian-economy-2/',
'https://insights.blackcoffer.com/impact-of-covid-19coronavirus-on-the-indian-economy/',
'https://insights.blackcoffer.com/impact-of-covid-19-coronavirus-on-the-global-economy/',
'https://insights.blackcoffer.com/ensuring-growth-through-insurance-technology/',
'https://insights.blackcoffer.com/blockchain-in-fintech/',
'https://insights.blackcoffer.com/blockchain-for-payments/',
'https://insights.blackcoffer.com/the-future-of-investing/',
'https://insights.blackcoffer.com/big-data-analytics-in-healthcare/',
'https://insights.blackcoffer.com/business-analytics-in-the-healthcare-industry/',
'https://insights.blackcoffer.com/challenges-and-opportunities-of-big-data-in-healthcare/',
'https://insights.blackcoffer.com/obstacles-to-data-driven-healthcare/',
'https://insights.blackcoffer.com/monetization-of-data-innovate-to-harvest-the-full-value-of-data/',
'https://insights.blackcoffer.com/traceability-of-information-master-your-data-capital/',
'https://insights.blackcoffer.com/bank-risk-management-india/',
'https://insights.blackcoffer.com/advance-analytics-for-refocusing-profits/',
'https://insights.blackcoffer.com/role-of-big-data-analytics-banking-and-finance/',
'https://insights.blackcoffer.com/mitigating-bank-risk-management/',
'https://insights.blackcoffer.com/future-of-bank-risk-management/',
'https://insights.blackcoffer.com/advanced-analytics-redefining-banking/',
'https://insights.blackcoffer.com/marketing-analytics-needs/',
'https://insights.blackcoffer.com/big-data-analytics-to-bring-transparency-and-good-governance/',
'https://insights.blackcoffer.com/how-political-leaders-will-shape-tomorrow-using-big-data-analytics/',
'https://insights.blackcoffer.com/big-data-and-analytics-to-help-form-political-leaders-win-election/',
'https://insights.blackcoffer.com/the-emergence-of-data-analytics/',
'https://insights.blackcoffer.com/how-artificial-intelligence-can-deliver-real-value-to-companies-2/',
'https://insights.blackcoffer.com/the-prospective-recipe-of-success-in-the-age-of-analytics/',
'https://insights.blackcoffer.com/using-people-analytics-to-drive-business-performance/',
'https://insights.blackcoffer.com/how-artificial-intelligence-can-deliver-real-value-to-companies/',
'https://insights.blackcoffer.com/big-data-analytics-through-iot-in-oil-and-gas-industry/',
'https://insights.blackcoffer.com/how-big-data-and-analytics-is-helping-marketing-leaders/',
'https://insights.blackcoffer.com/what-analytics-outsourcing-engagement-model-is-right-for-you/',
'https://insights.blackcoffer.com/big-data-analytics-solving-problems-banking-and-finance-industry/',
'https://insights.blackcoffer.com/analytics-healthcare-industry/',
'https://insights.blackcoffer.com/business-analytics-textile-industry/',
'https://insights.blackcoffer.com/big-data-marketing-reality/',
'https://insights.blackcoffer.com/understanding-millennial-market/',
'https://insights.blackcoffer.com/analytics-helping-fashion-e-tailers-develop-markets-developing-countries/',
'https://insights.blackcoffer.com/big-data-analytics-change-healthcare-developing-countries/',
'https://insights.blackcoffer.com/big-data-analytics-help-voters-know-political-leaders/',
'https://insights.blackcoffer.com/digital-transformation-oil-gas/',
'https://insights.blackcoffer.com/data-analytics-reduce-cost-production/',
'https://insights.blackcoffer.com/gaining-insights-internal-data-retail/',
'https://insights.blackcoffer.com/role-big-data-cyber-security/',
'https://insights.blackcoffer.com/role-big-data-healthcare/',
'https://insights.blackcoffer.com/data-driven-dashboards/',
'https://insights.blackcoffer.com/role-big-data-academia/',
'https://insights.blackcoffer.com/sales-forecasting-in-retail/',
'https://insights.blackcoffer.com/detect-data-exfiltration-over-the-network/',
'https://insights.blackcoffer.com/data-exfiltration/',
'https://insights.blackcoffer.com/impacts-of-covid-19-on-vegetable-vendors-and-food-stalls/']
time.sleep(2)
# Pass multiple urls using 'for' loop to 'chrome' driver
for i in range(len(urls)):
path =r"C:\Users\prvzs\Downloads\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(path,options=options)
driver.get(urls[i])
#driver.maximize_window()
# Get 'article_title' and 'article_paragraph' using xpath
article_title = driver.find_element_by_xpath('//h1').text
article_paragraph = driver.find_element_by_xpath("//div[contains(@class,'td-post-content')]").text
time.sleep(2)
# Save as '.txt' by naming 'URL_ID' from 1 to 171
for j in range(1,172,1):
with open(f'data/{j}.txt','w') as file:
file.write(article_title+"\n" )
file.writelines("% s" %data for data in article_paragraph)
# Quit driver
driver.quit()
这里是示例输出文件，包含所有 170 个 txt 保存文件中的所有相同数据。
下面两行：
# Get 'article_title' and 'article_paragraph' using xpath
article_title = driver.find_element_by_xpath('//h1').text
article_paragraph = driver.find_element_by_xpath("//div[contains(@class,'td-post-content')]").text
应该在第一个循环内。
在第一个循环内执行所有操作（甚至写入文件）。不需要第二个循环。
我有一组 urls 大约 170。所有 urls 来自同一个网站 www.blackoffer.com，它包含文章关于不同的主题。
我的任务是从页面中提取文章标题和段落。
我正在使用 selenium==3.141.0 来抓取数据。当我编写代码和 运行 '.py' 文件时。它 returns 所有 170 个文本文件（我想要的），但它 returns 所有 170 个文件中只有相同的数据。
这是我的代码
# Import required libraries
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
# Set options to display the window of chrome
options = Options()
options.headless = True
options.add_argument('window-size=1920x1080')
# Multiple urls
urls = ['https://insights.blackcoffer.com/how-is-login-logout-time-tracking-for-employees-in-office-done-by-ai/',
'https://insights.blackcoffer.com/how-does-ai-help-to-monitor-retail-shelf-watches/',
'https://insights.blackcoffer.com/ai-and-its-impact-on-the-fashion-industry/',
'https://insights.blackcoffer.com/how-do-deep-learning-models-predict-old-and-new-drugs-that-are-successfully-treated-in-healthcare/',
'https://insights.blackcoffer.com/how-artificial-intelligence-can-boost-your-productivity-level/',
'https://insights.blackcoffer.com/how-are-genetic-sequencing-maps-affected-by-deep-learning-and-ai/',
'https://insights.blackcoffer.com/how-is-ai-used-to-solve-traffic-management/',
'https://insights.blackcoffer.com/benefits-of-big-data-in-different-fields/',
'https://insights.blackcoffer.com/how-big-data-will-impact-the-future-of-business/',
'https://insights.blackcoffer.com/how-will-ai-make-decisions-in-tomorrows-wars/',
'https://insights.blackcoffer.com/which-one-is-better-ai-or-big-data/',
'https://insights.blackcoffer.com/how-robots-can-help-in-e-learning-platforms/',
'https://insights.blackcoffer.com/how-does-big-data-help-in-finance-and-the-growth-of-large-firms/',
'https://insights.blackcoffer.com/future-of-work-robot-ai-and-automation/',
'https://insights.blackcoffer.com/how-ai-will-help-the-defense-power-of-a-country/',
'https://insights.blackcoffer.com/future-of-ai-and-machine-roles-in-the-medical-sector/',
'https://insights.blackcoffer.com/ai-in-healthcare-to-improve-patient-outcomes/',
'https://insights.blackcoffer.com/what-if-the-creation-is-taking-over-the-creator/',
'https://insights.blackcoffer.com/what-jobs-will-robots-take-from-humans-in-the-future/',
'https://insights.blackcoffer.com/will-machine-replace-the-human-in-the-future-of-work/',
'https://insights.blackcoffer.com/will-ai-replace-us-or-work-with-us/',
'https://insights.blackcoffer.com/man-and-machines-together-machines-are-more-diligent-than-humans-blackcoffe/',
'https://insights.blackcoffer.com/in-future-or-in-upcoming-years-humans-and-machines-are-going-to-work-together-in-every-field-of-work/',
'https://insights.blackcoffer.com/how-neural-networks-can-be-applied-in-various-areas-in-the-future/',
'https://insights.blackcoffer.com/how-machine-learning-will-affect-your-business/',
'https://insights.blackcoffer.com/deep-learning-impact-on-areas-of-e-learning/',
'https://insights.blackcoffer.com/how-to-protect-future-data-and-its-privacy-blackcoffer/',
'https://insights.blackcoffer.com/how-machines-ai-automations-and-robo-human-are-effective-in-finance-and-banking/',
'https://insights.blackcoffer.com/ai-human-robotics-machine-future-planet-blackcoffer-thinking-jobs-workplace/',
'https://insights.blackcoffer.com/how-ai-will-change-the-world-blackcoffer/',
'https://insights.blackcoffer.com/future-of-work-how-ai-has-entered-the-workplace/',
'https://insights.blackcoffer.com/ai-tool-alexa-google-assistant-finance-banking-tool-future/',
'https://insights.blackcoffer.com/ai-healthcare-revolution-ml-technology-algorithm-google-analytics-industrialrevolution/',
'https://insights.blackcoffer.com/all-you-need-to-know-about-online-marketing/',
'https://insights.blackcoffer.com/evolution-of-advertising-industry/',
'https://insights.blackcoffer.com/how-data-analytics-can-help-your-business-respond-to-the-impact-of-covid-19/',
'https://insights.blackcoffer.com/covid-19-environmental-impact-for-the-future/',
'https://insights.blackcoffer.com/environmental-impact-of-the-covid-19-pandemic-lesson-for-the-future/',
'https://insights.blackcoffer.com/how-data-analytics-and-ai-are-used-to-halt-the-covid-19-pandemic/',
'https://insights.blackcoffer.com/difference-between-artificial-intelligence-machine-learning-statistics-and-data-mining/',
'https://insights.blackcoffer.com/how-python-became-the-first-choice-for-data-science/',
'https://insights.blackcoffer.com/how-google-fit-measure-heart-and-respiratory-rates-using-a-phone/',
'https://insights.blackcoffer.com/what-is-the-future-of-mobile-apps/',
'https://insights.blackcoffer.com/impact-of-ai-in-health-and-medicine/',
'https://insights.blackcoffer.com/telemedicine-what-patients-like-and-dislike-about-it/',
'https://insights.blackcoffer.com/how-we-forecast-future-technologies/',
'https://insights.blackcoffer.com/can-robots-tackle-late-life-loneliness/',
'https://insights.blackcoffer.com/embedding-care-robots-into-society-socio-technical-considerations/',
'https://insights.blackcoffer.com/management-challenges-for-future-digitalization-of-healthcare-services/',
'https://insights.blackcoffer.com/are-we-any-closer-to-preventing-a-nuclear-holocaust/',
'https://insights.blackcoffer.com/will-technology-eliminate-the-need-for-animal-testing-in-drug-development/',
'https://insights.blackcoffer.com/will-we-ever-understand-the-nature-of-consciousness/',
'https://insights.blackcoffer.com/will-we-ever-colonize-outer-space/',
'https://insights.blackcoffer.com/what-is-the-chance-homo-sapiens-will-survive-for-the-next-500-years/',
'https://insights.blackcoffer.com/why-does-your-business-need-a-chatbot/',
'https://insights.blackcoffer.com/how-you-lead-a-project-or-a-team-without-any-technical-expertise/',
'https://insights.blackcoffer.com/can-you-be-great-leader-without-technical-expertise/',
'https://insights.blackcoffer.com/how-does-artificial-intelligence-affect-the-environment/',
'https://insights.blackcoffer.com/how-to-overcome-your-fear-of-making-mistakes-2/',
'https://insights.blackcoffer.com/is-perfection-the-greatest-enemy-of-productivity/',
'https://insights.blackcoffer.com/global-financial-crisis-2008-causes-effects-and-its-solution/',
'https://insights.blackcoffer.com/gender-diversity-and-equality-in-the-tech-industry/',
'https://insights.blackcoffer.com/how-to-overcome-your-fear-of-making-mistakes/',
'https://insights.blackcoffer.com/how-small-business-can-survive-the-coronavirus-crisis/',
'https://insights.blackcoffer.com/impacts-of-covid-19-on-vegetable-vendors-and-food-stalls/',
'https://insights.blackcoffer.com/impacts-of-covid-19-on-vegetable-vendors/',
'https://insights.blackcoffer.com/impact-of-covid-19-pandemic-on-tourism-aviation-industries/',
'https://insights.blackcoffer.com/impact-of-covid-19-pandemic-on-sports-events-around-the-world/',
'https://insights.blackcoffer.com/changing-landscape-and-emerging-trends-in-the-indian-it-ites-industry/',
'https://insights.blackcoffer.com/online-gaming-adolescent-online-gaming-effects-demotivated-depression-musculoskeletal-and-psychosomatic-symptoms/',
'https://insights.blackcoffer.com/human-rights-outlook/',
'https://insights.blackcoffer.com/how-voice-search-makes-your-business-a-successful-business/',
'https://insights.blackcoffer.com/how-the-covid-19-crisis-is-redefining-jobs-and-services/',
'https://insights.blackcoffer.com/how-to-increase-social-media-engagement-for-marketers/',
'https://insights.blackcoffer.com/impacts-of-covid-19-on-streets-sides-food-stalls/',
'https://insights.blackcoffer.com/coronavirus-impact-on-energy-markets-2/',
'https://insights.blackcoffer.com/coronavirus-impact-on-the-hospitality-industry-5/',
'https://insights.blackcoffer.com/lessons-from-the-past-some-key-learnings-relevant-to-the-coronavirus-crisis-4/',
'https://insights.blackcoffer.com/estimating-the-impact-of-covid-19-on-the-world-of-work-2/',
'https://insights.blackcoffer.com/estimating-the-impact-of-covid-19-on-the-world-of-work-3/',
'https://insights.blackcoffer.com/travel-and-tourism-outlook/',
'https://insights.blackcoffer.com/gaming-disorder-and-effects-of-gaming-on-health/',
'https://insights.blackcoffer.com/what-is-the-repercussion-of-the-environment-due-to-the-covid-19-pandemic-situation/',
'https://insights.blackcoffer.com/what-is-the-repercussion-of-the-environment-due-to-the-covid-19-pandemic-situation-2/',
'https://insights.blackcoffer.com/impact-of-covid-19-pandemic-on-office-space-and-co-working-industries/',
'https://insights.blackcoffer.com/contribution-of-handicrafts-visual-arts-literature-in-the-indian-economy/',
'https://insights.blackcoffer.com/how-covid-19-is-impacting-payment-preferences/',
'https://insights.blackcoffer.com/how-will-covid-19-affect-the-world-of-work-2/',
'https://insights.blackcoffer.com/lessons-from-the-past-some-key-learnings-relevant-to-the-coronavirus-crisis/',
'https://insights.blackcoffer.com/covid-19-how-have-countries-been-responding/',
'https://insights.blackcoffer.com/coronavirus-impact-on-the-hospitality-industry-2/',
'https://insights.blackcoffer.com/how-will-covid-19-affect-the-world-of-work-3/',
'https://insights.blackcoffer.com/coronavirus-impact-on-the-hospitality-industry-3/',
'https://insights.blackcoffer.com/estimating-the-impact-of-covid-19-on-the-world-of-work/',
'https://insights.blackcoffer.com/covid-19-how-have-countries-been-responding-2/',
'https://insights.blackcoffer.com/how-will-covid-19-affect-the-world-of-work-4/',
'https://insights.blackcoffer.com/lessons-from-the-past-some-key-learnings-relevant-to-the-coronavirus-crisis-2/',
'https://insights.blackcoffer.com/lessons-from-the-past-some-key-learnings-relevant-to-the-coronavirus-crisis-3/',
'https://insights.blackcoffer.com/coronavirus-impact-on-the-hospitality-industry-4/',
'https://insights.blackcoffer.com/why-scams-like-nirav-modi-happen-with-indian-banks/',
'https://insights.blackcoffer.com/impact-of-covid-19-on-the-global-economy/',
'https://insights.blackcoffer.com/impact-of-covid-19coronavirus-on-the-indian-economy-2/',
'https://insights.blackcoffer.com/impact-of-covid-19-on-the-global-economy-2/',
'https://insights.blackcoffer.com/impact-of-covid-19-coronavirus-on-the-indian-economy-3/',
'https://insights.blackcoffer.com/should-celebrities-be-allowed-to-join-politics/',
'https://insights.blackcoffer.com/how-prepared-is-india-to-tackle-a-possible-covid-19-outbreak/',
'https://insights.blackcoffer.com/how-will-covid-19-affect-the-world-of-work/',
'https://insights.blackcoffer.com/controversy-as-a-marketing-strategy/',
'https://insights.blackcoffer.com/coronavirus-impact-on-the-hospitality-industry/',
'https://insights.blackcoffer.com/coronavirus-impact-on-energy-markets/',
'https://insights.blackcoffer.com/what-are-the-key-policies-that-will-mitigate-the-impacts-of-covid-19-on-the-world-of-work/',
'https://insights.blackcoffer.com/marketing-drives-results-with-a-focus-on-problems/',
'https://insights.blackcoffer.com/continued-demand-for-sustainability/',
'https://insights.blackcoffer.com/coronavirus-disease-covid-19-effect-the-impact-and-role-of-mass-media-during-the-pandemic/',
'https://insights.blackcoffer.com/should-people-wear-fabric-gloves-seeking-evidence-regarding-the-differential-transfer-of-covid-19-or-coronaviruses-generally-between-surfaces/',
'https://insights.blackcoffer.com/why-is-there-a-severe-immunological-and-inflammatory-explosion-in-those-affected-by-sarms-covid-19/',
'https://insights.blackcoffer.com/what-do-you-think-is-the-lesson-or-lessons-to-be-learned-with-covid-19/',
'https://insights.blackcoffer.com/coronavirus-the-unexpected-challenge-for-the-european-union/',
'https://insights.blackcoffer.com/industrial-revolution-4-0-pros-and-cons/',
'https://insights.blackcoffer.com/impact-of-covid-19-coronavirus-on-the-indian-economy/',
'https://insights.blackcoffer.com/impact-of-covid-19-coronavirus-on-the-indian-economy-2/',
'https://insights.blackcoffer.com/impact-of-covid-19coronavirus-on-the-indian-economy/',
'https://insights.blackcoffer.com/impact-of-covid-19-coronavirus-on-the-global-economy/',
'https://insights.blackcoffer.com/ensuring-growth-through-insurance-technology/',
'https://insights.blackcoffer.com/blockchain-in-fintech/',
'https://insights.blackcoffer.com/blockchain-for-payments/',
'https://insights.blackcoffer.com/the-future-of-investing/',
'https://insights.blackcoffer.com/big-data-analytics-in-healthcare/',
'https://insights.blackcoffer.com/business-analytics-in-the-healthcare-industry/',
'https://insights.blackcoffer.com/challenges-and-opportunities-of-big-data-in-healthcare/',
'https://insights.blackcoffer.com/obstacles-to-data-driven-healthcare/',
'https://insights.blackcoffer.com/monetization-of-data-innovate-to-harvest-the-full-value-of-data/',
'https://insights.blackcoffer.com/traceability-of-information-master-your-data-capital/',
'https://insights.blackcoffer.com/bank-risk-management-india/',
'https://insights.blackcoffer.com/advance-analytics-for-refocusing-profits/',
'https://insights.blackcoffer.com/role-of-big-data-analytics-banking-and-finance/',
'https://insights.blackcoffer.com/mitigating-bank-risk-management/',
'https://insights.blackcoffer.com/future-of-bank-risk-management/',
'https://insights.blackcoffer.com/advanced-analytics-redefining-banking/',
'https://insights.blackcoffer.com/marketing-analytics-needs/',
'https://insights.blackcoffer.com/big-data-analytics-to-bring-transparency-and-good-governance/',
'https://insights.blackcoffer.com/how-political-leaders-will-shape-tomorrow-using-big-data-analytics/',
'https://insights.blackcoffer.com/big-data-and-analytics-to-help-form-political-leaders-win-election/',
'https://insights.blackcoffer.com/the-emergence-of-data-analytics/',
'https://insights.blackcoffer.com/how-artificial-intelligence-can-deliver-real-value-to-companies-2/',
'https://insights.blackcoffer.com/the-prospective-recipe-of-success-in-the-age-of-analytics/',
'https://insights.blackcoffer.com/using-people-analytics-to-drive-business-performance/',
'https://insights.blackcoffer.com/how-artificial-intelligence-can-deliver-real-value-to-companies/',
'https://insights.blackcoffer.com/big-data-analytics-through-iot-in-oil-and-gas-industry/',
'https://insights.blackcoffer.com/how-big-data-and-analytics-is-helping-marketing-leaders/',
'https://insights.blackcoffer.com/what-analytics-outsourcing-engagement-model-is-right-for-you/',
'https://insights.blackcoffer.com/big-data-analytics-solving-problems-banking-and-finance-industry/',
'https://insights.blackcoffer.com/analytics-healthcare-industry/',
'https://insights.blackcoffer.com/business-analytics-textile-industry/',
'https://insights.blackcoffer.com/big-data-marketing-reality/',
'https://insights.blackcoffer.com/understanding-millennial-market/',
'https://insights.blackcoffer.com/analytics-helping-fashion-e-tailers-develop-markets-developing-countries/',
'https://insights.blackcoffer.com/big-data-analytics-change-healthcare-developing-countries/',
'https://insights.blackcoffer.com/big-data-analytics-help-voters-know-political-leaders/',
'https://insights.blackcoffer.com/digital-transformation-oil-gas/',
'https://insights.blackcoffer.com/data-analytics-reduce-cost-production/',
'https://insights.blackcoffer.com/gaining-insights-internal-data-retail/',
'https://insights.blackcoffer.com/role-big-data-cyber-security/',
'https://insights.blackcoffer.com/role-big-data-healthcare/',
'https://insights.blackcoffer.com/data-driven-dashboards/',
'https://insights.blackcoffer.com/role-big-data-academia/',
'https://insights.blackcoffer.com/sales-forecasting-in-retail/',
'https://insights.blackcoffer.com/detect-data-exfiltration-over-the-network/',
'https://insights.blackcoffer.com/data-exfiltration/',
'https://insights.blackcoffer.com/impacts-of-covid-19-on-vegetable-vendors-and-food-stalls/']
time.sleep(2)
# Pass multiple urls using 'for' loop to 'chrome' driver
for i in range(len(urls)):
path =r"C:\Users\prvzs\Downloads\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(path,options=options)
driver.get(urls[i])
#driver.maximize_window()
# Get 'article_title' and 'article_paragraph' using xpath
article_title = driver.find_element_by_xpath('//h1').text
article_paragraph = driver.find_element_by_xpath("//div[contains(@class,'td-post-content')]").text
time.sleep(2)
# Save as '.txt' by naming 'URL_ID' from 1 to 171
for j in range(1,172,1):
with open(f'data/{j}.txt','w') as file:
file.write(article_title+"\n" )
file.writelines("% s" %data for data in article_paragraph)
# Quit driver
driver.quit()
这里是示例输出文件，包含所有 170 个 txt 保存文件中的所有相同数据。
下面两行：
# Get 'article_title' and 'article_paragraph' using xpath
article_title = driver.find_element_by_xpath('//h1').text
article_paragraph = driver.find_element_by_xpath("//div[contains(@class,'td-post-content')]").text
应该在第一个循环内。
在第一个循环内执行所有操作（甚至写入文件）。不需要第二个循环。