写入硒数据时为空 CSV
Empty CSV when writing selenium data
我正在第一次体验 Selenium 并在一个知名的视频平台上做一些教程。它在大多数时候都非常可靠。但是,我遇到了一些创建 CSV 但未导出数据的页面的问题。 CSV 被“触摸”,但它不导出正常打印中显示的数据。
谁能帮我解决这个脚本的问题?
#_*_coding: utf-8_*_
from selenium import webdriver
import selenium
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import csv
import os
os.chdir("C:\Selenium")
PATH = "chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://twitter-trends.iamrohit.in/")
try:
main = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CLASS_NAME, "panel-body"))
)
main = (main.text)
f = open('twitter.csv', 'wb')
print(main, file = f)
f.close()
#print(main)
except:
driver.quit()
driver.quit()
Python 版本 3.7.4,Selenium 版本 3.141.0,Windows10
正在调试您的代码,您正在正确获取数据。
main = (main.text)
print(main)
f = open('twitter.csv', 'wb')
所以错误是在写入输出文件时出现的。替换
的代码
main = (main.text)
with open('twitter.txt', 'wb', encoding='utf-8') as file1:
# Writing data to a file
file1.writelines(main)
会工作,如果你检查打印你有中文字符,这将使输出文件中的写入失败。
要抓取 Twitter Trends - Worldwide table you can use DataFrame from Python Pandas and write it to a csv file using the following :
代码块:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
driver.get("https://twitter-trends.iamrohit.in/")
driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//b[text()='Note:']"))))
headers = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "thead > tr > th")))]
ranks = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='twitter-trends']//tbody//tr//descendant::th[1]")))]
topics = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='twitter-trends']//tbody/tr//descendant::th[2]/a")))]
volumes = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='twitter-trends']//tbody/tr//descendant::th[3]")))]
df = pd.DataFrame(data=list(zip(ranks, topics, volumes)), columns=headers)
df.to_csv(r'C:\Data_Files\output_files\twitter.csv', index=False)
driver.quit()
CSV 快照:
参考资料
您可以在以下位置找到一些相关的详细讨论:
- Selenium: Web-Scraping Historical Data from Coincodex and transform into a Pandas Dataframe
我正在第一次体验 Selenium 并在一个知名的视频平台上做一些教程。它在大多数时候都非常可靠。但是,我遇到了一些创建 CSV 但未导出数据的页面的问题。 CSV 被“触摸”,但它不导出正常打印中显示的数据。
谁能帮我解决这个脚本的问题?
#_*_coding: utf-8_*_
from selenium import webdriver
import selenium
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import csv
import os
os.chdir("C:\Selenium")
PATH = "chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://twitter-trends.iamrohit.in/")
try:
main = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CLASS_NAME, "panel-body"))
)
main = (main.text)
f = open('twitter.csv', 'wb')
print(main, file = f)
f.close()
#print(main)
except:
driver.quit()
driver.quit()
Python 版本 3.7.4,Selenium 版本 3.141.0,Windows10
正在调试您的代码,您正在正确获取数据。
main = (main.text)
print(main)
f = open('twitter.csv', 'wb')
所以错误是在写入输出文件时出现的。替换
的代码main = (main.text)
with open('twitter.txt', 'wb', encoding='utf-8') as file1:
# Writing data to a file
file1.writelines(main)
会工作,如果你检查打印你有中文字符,这将使输出文件中的写入失败。
要抓取 Twitter Trends - Worldwide table you can use DataFrame from Python Pandas and write it to a csv file using the following
代码块:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
driver.get("https://twitter-trends.iamrohit.in/")
driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//b[text()='Note:']"))))
headers = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "thead > tr > th")))]
ranks = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='twitter-trends']//tbody//tr//descendant::th[1]")))]
topics = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='twitter-trends']//tbody/tr//descendant::th[2]/a")))]
volumes = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='twitter-trends']//tbody/tr//descendant::th[3]")))]
df = pd.DataFrame(data=list(zip(ranks, topics, volumes)), columns=headers)
df.to_csv(r'C:\Data_Files\output_files\twitter.csv', index=False)
driver.quit()
CSV 快照:
参考资料
您可以在以下位置找到一些相关的详细讨论:
- Selenium: Web-Scraping Historical Data from Coincodex and transform into a Pandas Dataframe