Selenium (python):未检索下一页显示的 HTML 中的文本

Selenium (python): not retrieving text in HTML displayed on next pages

我正试图从我们的国家药品监督管理局收集药品、制造商和 auth.codes (AIC) 的名称。

下面的代码几乎可以工作,但只捕获网页上显示的文本,而不是我在 HTML 中看到但会显示在后续页面上的文本:

所以在下面的示例搜索中,我将药物打印到 ABACUS 药物名称,然后打印空行。我究竟做错了什么?提前谢谢你。

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome(
    "/Users/bob/Documents/work/AIFA/scraper/scrape_gu/chromedriver"
)

# start from AIFA's search page and automate steps to drug search function
driver.get("https://farmaci.agenziafarmaco.gov.it/bancadatifarmaci/cerca-farmaco")
lettocompreso = driver.find_element_by_id("conf")
lettocompreso.click()
accetta = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, "/html/body/div[5]/div[3]/div/button"))
)
accetta.click()

# now click on the search button with drug (fragment) name
cerca_textbox = driver.find_element_by_id("search")
cerca_textbox.send_keys("AB")
cerca_textbox.send_keys(Keys.ENTER)
# now parse the results
toplistafarmaci = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.XPATH, '//*[@id="ul_farm_results"]/li'))
)

# now all data I need are in <span> triples
drugs = []

for drug in driver.find_elements_by_tag_name("span"):
    print(drug.text)

当然,您可以完全忽略代码,直到出现“现在分析结果”注释行,该注释行仅用于跳转到实际搜索。

如果您只是想从所有页面中提取数据,那么您需要进行以下更改,例如使用此 xpath //*[@id='ul_farm_results']/li[@style='display: list-item;'] 并将所有内容存储在一个列表中,然后在所有页面上迭代该列表页数。

cerca_textbox = driver.find_element_by_id("search")
cerca_textbox.send_keys("AB")
cerca_textbox.send_keys(Keys.ENTER)
# now parse the results
# now all data I need are in <span> triples
drugs = []

wait = WebDriverWait(driver, 30)
total_number_of_pages = 7

while total_number_of_pages > 0:
    try:
        toplistafarmaci = wait.until(EC.presence_of_all_elements_located((By.XPATH, "//*[@id='ul_farm_results']/li[@style='display: list-item;']")))
        for drug in toplistafarmaci:
            print(drug.text)
            #drugs.append(drug.text)
        wait.until(EC.element_to_be_clickable((By.XPATH, "//button[text()='>']"))).click()
        time.sleep(2)
        print('Clicked on next page')
        total_number_of_pages = total_number_of_pages - 1
    except:
        print('Could not click on next page arrow')
        break

输出:

AIC: 045359
ABACAVIR E LAMIVUDINA ACCORD
Azienda: ACCORD HEALTHCARE, S.L.U.
AIC: 045522
ABACAVIR E LAMIVUDINA ARISTO
Azienda: ARISTO PHARMA GMBH
AIC: 044669
ABACAVIR E LAMIVUDINA AUROBINDO
Azienda: AUROBINDO PHARMA (ITALIA) S.R.L.
AIC: 044114
ABACAVIR E LAMIVUDINA DOC GENERICI
Azienda: DOC GENERICI SRL
AIC: 044980
ABACAVIR E LAMIVUDINA DR. REDDY'S
Azienda: DR. REDDY'S S.R.L.
AIC: 044719
ABACAVIR E LAMIVUDINA EG
Azienda: EG S.P.A.
AIC: 043972
ABACAVIR E LAMIVUDINA LUPIN
Azienda: LUPIN (EUROPE) LIMITED
AIC: 043971
ABACAVIR E LAMIVUDINA MYLAN
Azienda: MYLAN S.P.A.
AIC: 044773
ABACAVIR E LAMIVUDINA MYLAN PHARMA
Azienda: MYLAN S.P.A.
AIC: 044838
ABACAVIR E LAMIVUDINA SANDOZ
Azienda: SANDOZ S.P.A.
AIC: 045348
ABACAVIR E LAMIVUDINA SUN
Azienda: SUN PHARMACEUTICAL INDUSTRIES (EUROPE) B.V.
AIC: 044286
ABACAVIR E LAMIVUDINA TEVA
Azienda: TEVA ITALIA S.R.L.
AIC: 045021
ABACAVIR E LAMIVUDINA ZENTIVA
Azienda: ZENTIVA ITALIA S.R.L.
AIC: 045354
ABACAVIR MYLAN
Azienda: MYLAN S.P.A.
AIC: 042645
ABACAVIR/LAMIVUDINA/ZIDOVUDINA MYLAN
Azienda: MYLAN S.P.A.
AIC: 028508
ABACUS
Azienda: DOMPE' FARMACEUTICI S.P.A.
Clicked on next page
AIC: 022773
ABADOX
Azienda: LA FAR SRL
AIC: 025605
ABAPRIM
Azienda: ISTITUTO GENTILI S.P.A.
AIC: 043658
ABASAGLAR
Azienda: ELI LILLY NEDERLAND B.V.
AIC: 036816
ABBA
Azienda: FIDIA FARMACEUTICI S.P.A.
AIC: 033002
ABELCET COMPLESSO LIPIDICO
Azienda: TEVA ITALIA S.R.L.
AIC: 044868
ABERIPRA
Azienda: ECUPHARMA S.R.L.
AIC: 040915
ABESART
Azienda: GENETIC S.P.A.
AIC: 024480
ABETOL
Azienda: LABORATORIO FARMACEUTICO C.T. S.R.L.
AIC: 049452
ABEVMY
Azienda: MYLAN IRE HEALTHCARE LIMITED
AIC: 026465
ABEX
Azienda: GLORIA MED PHARMA S.R.L.
AIC: 004097
ABIDEC
Azienda: TEOFARMA S.R.L.
AIC: 039371
ABIDOL
Azienda: AURORA LICENSING SRL
AIC: 047244
ABIES LINDA'S
Azienda: LABORATORI OMEOPATICI LINDA'S S.R.L.
AIC: 046630
ABIES NIGRA
Azienda: BOIRON
AIC: 027142
ABILENE
Azienda: FARMACEUTICI FORMENTI S.P.A.
AIC: 036582
ABILIFY
Azienda: OTSUKA PHARMACEUTICAL NETHERLANDS BV
Clicked on next page
AIC: 043143
ABILIFY MAINTENA
Azienda: OTSUKA PHARMACEUTICAL NETHERLANDS BV
AIC: 025358
ABIMONO
Azienda: FARMITALIA INDUSTRIA CHIMICO FARMACEUTICA S.R.L.
AIC: 038637
ABINIX
Azienda: GENETIC S.P.A.
AIC: 024912
ABIOCEF
Azienda: NUOVO CONSORZIO SANITARIO NAZIONALE
AIC: 037350
ABIOCLAV
Azienda: AESCULAPIUS FARMACEUTICI SRL
AIC: 010772
ABIOSTIL
Azienda: LABORATORIO CHIMICO DECA DR.CAPUANI S.R.L.
AIC: 049479
ABIRATERONE ACCORD
Azienda: ACCORD HEALTHCARE, S.L.U.
AIC: 049583
ABIRATERONE KRKA
Azienda: KRKA D.D. NOVO MESTO
AIC: 049686
ABIRATERONE MYLAN
Azienda: MYLAN IRELAND LIMITED
AIC: 049298
ABIRATERONE TIEFENBACHER
Azienda: ALFRED E. TIEFENBACHER GMBH & CO. KG
AIC: 038038
ABIS
Azienda: A.G.I.P.S. FARMACEUTICI SRL
AIC: 021826
ABLACTON
Azienda: SCHERING SPA
AIC: 039399
ABRAXANE
Azienda: BRISTOL-MYERS SQUIBB PHARMA EEIG
AIC: 042292
ABRIFF
Azienda: MUNDIPHARMA PHARMACEUTICALS S.R.L.
AIC: 047231
ABROSYX N
Azienda: HOMEOSYN ITALIA S.R.L.
AIC: 046639
ABROTANUM
Azienda: BOIRON
Clicked on next page
AIC: 038494
ABSEAMED
Azienda: MEDICE ARZNEIMITTEL PUETTER GMBH & CO.KG
AIC: 036018
ABSORCOL
Azienda: ADDENDA PHARMA S.R.L.
AIC: 038736
ABSTRAL
Azienda: KYOWA KIRIN HOLDINGS B.V.
AIC: 033482
ACICLOVIR ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 025105
AMBROXOLO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035223
AMOXICILLINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 036819
AMOXICILLINA E ACIDO CLAVULANICO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 033378
ATENOLOLO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 040908
ATORVASTATINA ABC
Azienda: PHARMACARE S.R.L.
AIC: 038758
AZITROMICINA ABC
Azienda: IPSO PHARMA S.R.L.
AIC: 034350
BACAMPICILLINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 039319
BROMAZEPAM ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035510
CAPTOPRIL ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 036294
CARBOCISTEINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035361
CEFACLORO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035235
CEFONICID ABC
Azienda: ABC FARMACEUTICI S.P.A.
Clicked on next page
AIC: 035375
CEFOTAXIMA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035924
CEFTRIAXONE ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 037911
CETIRIZINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 037929
CIPROFLOXACINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 036043
CITALOPRAM ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035129
CLODRONATO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 036240
DIAZEPAM ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035809
DOMPERIDONE ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 033876
DOXOFILLINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 038414
FINASTERIDE ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 037931
FLUCONAZOLO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 036519
GABAPENTIN ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 036277
GENTAMICINA E BETAMETASONE ABC
Azienda: LABORATORIO FARMACOLOGICO MILANESE S.R.L.
AIC: 034584
GLUCONATO FERROSO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 044042
IBUPROFENE ABC FARMACEUTICI
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 038358
KETOROLAC ABC
Azienda: ABC FARMACEUTICI S.P.A.
Clicked on next page
AIC: 038064
LANSOPRAZOLO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 039217
LATTULOSIO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 040046
LEVOFLOXACINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035931
LORAZEPAM ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 039304
LORMETAZEPAM ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 042794
MEMANTINA ABDI
Azienda: ABDI FARMA UNIPESSOAL LDA
AIC: 037852
METAMIZOLO SODICO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 039807
NEBIVOLOLO ABC
Azienda: PHARMACARE S.R.L.
AIC: 025000
NORFLOXACINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 040340
OLANZAPINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 037665
OMEPRAZOLO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 041022
PANTOPRAZOLO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 025103
PIROXICAM ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 031837
PROPAFENONE ABBOTT
Azienda: ABBOTT S.R.L.
AIC: 038272
RAMIPRIL ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035505
RANITIDINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
Clicked on next page
AIC: 037278
RIFAXIMINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 043975
ROSUVASTATINA ABC
Azienda: IPSO PHARMA S.R.L.
AIC: 041742
SILDENAFIL ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 037352
SIMVASTATINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035134
SUCRALFATO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 045643
TADALAFIL ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 036950
TAMSULOSINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 037941
TRAMADOLO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 018527
UREA ABBOTT
Azienda: ABBOTT S.R.L.
AIC: 040762
VALSARTAN ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 039486
VENLAFAXINA ABC
Azienda: BLUEFISH PHARMACEUTICALS AB
AIC: 026179
VERAPAMIL ABBOTT
Azienda: ABBOTT S.R.L.
Clicked on next page

Process finished with exit code 0