写入硒数据时为空 CSV

Empty CSV when writing selenium data

我正在第一次体验 Selenium 并在一个知名的视频平台上做一些教程。它在大多数时候都非常可靠。但是,我遇到了一些创建 CSV 但未导出数据的页面的问题。 CSV 被“触摸”,但它不导出正常打印中显示的数据。

谁能帮我解决这个脚本的问题?

#_*_coding: utf-8_*_


from selenium import webdriver
import selenium
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import csv
import os

os.chdir("C:\Selenium")
PATH = "chromedriver.exe"
driver = webdriver.Chrome(PATH)

driver.get("https://twitter-trends.iamrohit.in/")

try: 
    main = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, "panel-body"))
    )
 
    main = (main.text)
 
    f = open('twitter.csv', 'wb')
    print(main, file = f)
    f.close()
    
    #print(main)

except:
    driver.quit()

driver.quit()

Python 版本 3.7.4,Selenium 版本 3.141.0,Windows10

正在调试您的代码,您正在正确获取数据。

   main = (main.text)
   print(main)
   f = open('twitter.csv', 'wb')

所以错误是在写入输出文件时出现的。替换

的代码
main = (main.text)

with open('twitter.txt', 'wb', encoding='utf-8') as file1:
# Writing data to a file
    file1.writelines(main)

会工作,如果你检查打印你有中文字符,这将使输出文件中的写入失败。

要抓取 Twitter Trends - Worldwide table you can use DataFrame from Python Pandas and write it to a csv file using the following

代码块:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

driver.get("https://twitter-trends.iamrohit.in/")
driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//b[text()='Note:']"))))
headers = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "thead > tr > th")))]
ranks = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='twitter-trends']//tbody//tr//descendant::th[1]")))]
topics = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='twitter-trends']//tbody/tr//descendant::th[2]/a")))]
volumes = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='twitter-trends']//tbody/tr//descendant::th[3]")))]
df = pd.DataFrame(data=list(zip(ranks, topics, volumes)), columns=headers)
df.to_csv(r'C:\Data_Files\output_files\twitter.csv', index=False)
driver.quit()

CSV 快照:


参考资料

您可以在以下位置找到一些相关的详细讨论:

  • Selenium: Web-Scraping Historical Data from Coincodex and transform into a Pandas Dataframe