如何从硒中的 console.log 中提取数组？
How can I extract an array from the console.log in selenium?
当我尝试使用
获取 console.log 时
driver.execute_script('values = [];Highcharts.charts[0].series[0].data.forEach(function(d){ values.push(d.y) });console.log(values);')
time.sleep(4)
for entry in driver.get_log('browser'):
print(entry)
我只是得到了控制台输出，但没有得到数组的实际内容，应该是这样的
screenhot of array
相反，我得到：
{'level': 'INFO', 'message': 'console-api 2:121 Array(261)', 'source': 'console-api', 'timestamp': 1647448899327}
我找到了一些关于如何解决这个问题的建议，但不幸的是，none 其中对我有用。有没有人有办法解决吗？
非常感谢！
编辑：
我找到了这个脚本 (https://bgrins.github.io/devtools-snippets/#console-save)，当我在浏览器中手动执行这个脚本时，它会将控制台输出保存到一个文件中，但是当我尝试在 selenium 中实现它时，比如
driver.execute_script('(function(console){console.save=function(data,filename){if(!data){console.error("Console.save: No data")return}if(!filename)filename="console.json"if(typeof data==="object"){data=JSON.stringify(data,undefined,4)}var blob=new Blob([data],{type:"text/json"}),e=document.createEvent("MouseEvents"),a=document.createElement("a")a.download=filename a.href=window.URL.createObjectURL(blob)a.dataset.downloadurl=["text/json",a.download,a.href].join(":")e.initMouseEvent("click",true,false,window,0,0,0,0,0,false,false,false,false,0,null)a.dispatchEvent(e)}})(console)')
driver.execute_script('values = [];Highcharts.charts[0].series[0].data.forEach(function(d){ values.push(d.y) });console.save(values, add);')
我收到这样的回复：
[19980:18944:0316/183240.115:ERROR:database.cc(1777)] OptOutBlacklist SQLite error: code 13 errno 112: database or disk is full sql: CREATE TABLE IF NOT EXISTS previews_v1 (host_name VARCHAR NOT NULL, time INTEGER NOT NULL, opt_out INTEGER NOT
NULL, type INTEGER NOT NULL, PRIMARY KEY(host_name, time DESC, opt_out, type))
[19980:18944:0316/183240.121:ERROR:database.cc(1777)] OptOutBlacklist SQLite error: code 1 errno 112: no such table: enabled_previews_v1 sql: SELECT type, version FROM enabled_previews_v1
[19980:18944:0316/183240.132:ERROR:database.cc(1777)] OptOutBlacklist SQLite error: code 1 errno 112: no such table: enabled_previews_v1 sql: INSERT INTO enabled_previews_v1 (type, version) VALUES (?, ?)
[19980:18944:0316/183240.180:ERROR:database.cc(1777)] OptOutBlacklist SQLite error: code 1 errno 112: no such table: previews_v1 sql: SELECT host_name, time, opt_out, type FROM previews_v1 ORDER BY host_name, time DESC
[15816:23948:0316/183240.237:ERROR:entry_impl.cc(999)] Failed to save user data
[15816:23948:0316/183240.259:ERROR:entry_impl.cc(999)] Failed to save user data
[15816:23948:0316/183240.340:ERROR:entry_impl.cc(999)] Failed to save user data
[15816:23948:0316/183240.359:ERROR:entry_impl.cc(999)] Failed to save user data
[19980:18944:0316/183240.607:ERROR:database.cc(1777)] Shortcuts SQLite error: code 13 errno 112: database or disk is full sql: COMMIT
[15816:23948:0316/183240.677:ERROR:entry_impl.cc(999)] Failed to save user data
[15816:23948:0316/183240.755:ERROR:entry_impl.cc(999)] Failed to save user data
[15816:23948:0316/183240.765:ERROR:entry_impl.cc(999)] Failed to save user data
[19980:18944:0316/183240.825:ERROR:database.cc(1777)] Shortcuts SQLite error: code 1 errno 112: no such table: omni_box_shortcuts sql: SELECT id, text, fill_into_edit, url, document_type, contents, contents_class, description, description_class, transition, type, keyword, last_access_time, number_of_hits FROM omni_box_shortcuts
[19980:19372:0316/183241.460:ERROR:database.cc(1777)] TopSites SQLite error: code 13 errno 0: database or disk is full sql: INSERT OR REPLACE INTO top_sites (url, url_rank, title) VALUES (?, ?, ?)
[19980:19372:0316/183241.468:ERROR:database.cc(1777)] TopSites SQLite error: code 13 errno 0: database or disk is full sql: INSERT OR REPLACE INTO top_sites (url, url_rank, title) VALUES (?, ?, ?)
[19980:19372:0316/183241.479:ERROR:database.cc(1777)] TopSites SQLite error: code 13 errno 0: database or disk is full sql: UPDATE top_sites SET url_rank = ? WHERE url == ?
[19980:19372:0316/183241.479:ERROR:database.cc(1777)] TopSites SQLite error: code 1 errno 0: cannot commit - no transaction is active sql: COMMIT
[19980:18944:0316/183241.550:ERROR:database.cc(1777)] Predictor SQLite error: code 13 errno 112: database or disk is full sql: COMMIT
我基本上用额外的 js 修复了它
driver.execute_script("""(function(console){
console.save = function(data, filename){
if(!data) {
console.error('Console.save: No data')
return;
}
if(!filename) filename = 'console.json'
if(typeof data === "object"){
data = JSON.stringify(data, undefined, 4)
}
var blob = new Blob([data], {type: 'text/json'}),
e = document.createEvent('MouseEvents'),
a = document.createElement('a')
a.download = filename
a.href = window.URL.createObjectURL(blob)
a.dataset.downloadurl = ['text/json', a.download, a.href].join(':')
e.initMouseEvent('click', true, false, window, 0, 0, 0, 0, 0, false, false, false, false, 0, null)
a.dispatchEvent(e)
}
})(console)""")
driver.execute_script('values = [];Highcharts.charts[0].series[0].data.forEach(function(d){ values.push(d.y) });')
driver.execute_script('console.save(values, add);')
这会将数组保存到 json
我认为是XYProblem。
你问如何从 console.log() 获取输出，但似乎真正的问题是如何从 Highcharts 获取数据并在 Python 中使用 - 而这不需要 console.log().
你可以在JavaScript中使用return values在Python中得到它。
它甚至不需要 JavaScript 中的 JSON.stringify() 和 Python 中的 json.loads()。
data = driver.execute_script('''
values = [];
Highcharts.charts[0].series[0].data.forEach((d) => values.push(d.y));
return values;
''')
print(data)
编辑：
使用 .map() 而不是 forEarch() 你可以将 JavaScript 减少到一行：
return Highcharts.charts[0].series[0].data.map(d => d.y);
https://www.highcharts.com/demo/line-basic
的最小工作示例
from selenium import webdriver
#from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.firefox import GeckoDriverManager
import time
url = 'https://www.highcharts.com/demo/line-basic'
#driver = webdriver.Chrome(executable_path=ChromeDriverManager().install())
driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
driver.get(url)
time.sleep(5)
data = driver.execute_script('''
values = [];
Highcharts.charts[0].series[0].data.forEach((d) => values.push(d.y));
return values;
''')
print(type(data), data)
# or shorter
data = driver.execute_script('return Highcharts.charts[0].series[0].data.map(d => d.y);')
print(type(data), data)
结果：
<class 'list'> [43934, 52503, 57177, 69658, 97031, 119931, 137133, 154175]
编辑：
例如，它可用于将带有时间戳（以毫秒为单位）的 x 转换为 MM/DD/YY 或 YYYY-MM-DD
等日期
import datetime
data = [1647212400000, 1647298800000, 1647385200000, 1647471600000]
new_data = []
for x in data:
dt = datetime.datetime.fromtimestamp(x//1000)
new_data.append(d.strftime('%D'))
print(d.strftime('%D'), '|', d.strftime('%Y-%m-%d'))
print('new_data:', new_data)
结果：
03/17/22 | 2022-03-17
03/17/22 | 2022-03-17
03/17/22 | 2022-03-17
03/17/22 | 2022-03-17
new_data: ['03/17/22', '03/17/22', '03/17/22', '03/17/22']
如果您将数据保存在 pandas.DataFrame 中，那么您可以使用 pd.to_datetime() 和 .dt.strftime('%D') 但是为了计算，最好将其保存为 datetime
import pandas as pd
data = [1647212400000, 1647298800000, 1647385200000, 1647471600000]
df = pd.DataFrame({'date': data})
print(df)
df['date'] = pd.to_datetime(df['date'], unit='ms')
df['date'] = df['date'].dt.strftime('%D')
print(df)
结果：
date
0 1647212400000
1 1647298800000
2 1647385200000
3 1647471600000
date
0 03/13/22
1 03/14/22
2 03/15/22
3 03/16/22
当我尝试使用
获取 console.log 时driver.execute_script('values = [];Highcharts.charts[0].series[0].data.forEach(function(d){ values.push(d.y) });console.log(values);')
time.sleep(4)
for entry in driver.get_log('browser'):
print(entry)
我只是得到了控制台输出，但没有得到数组的实际内容，应该是这样的 screenhot of array
相反，我得到：
{'level': 'INFO', 'message': 'console-api 2:121 Array(261)', 'source': 'console-api', 'timestamp': 1647448899327}
我找到了一些关于如何解决这个问题的建议，但不幸的是，none 其中对我有用。有没有人有办法解决吗？ 非常感谢！
编辑： 我找到了这个脚本 (https://bgrins.github.io/devtools-snippets/#console-save)，当我在浏览器中手动执行这个脚本时，它会将控制台输出保存到一个文件中，但是当我尝试在 selenium 中实现它时，比如
driver.execute_script('(function(console){console.save=function(data,filename){if(!data){console.error("Console.save: No data")return}if(!filename)filename="console.json"if(typeof data==="object"){data=JSON.stringify(data,undefined,4)}var blob=new Blob([data],{type:"text/json"}),e=document.createEvent("MouseEvents"),a=document.createElement("a")a.download=filename a.href=window.URL.createObjectURL(blob)a.dataset.downloadurl=["text/json",a.download,a.href].join(":")e.initMouseEvent("click",true,false,window,0,0,0,0,0,false,false,false,false,0,null)a.dispatchEvent(e)}})(console)')
driver.execute_script('values = [];Highcharts.charts[0].series[0].data.forEach(function(d){ values.push(d.y) });console.save(values, add);')
我收到这样的回复：
[19980:18944:0316/183240.115:ERROR:database.cc(1777)] OptOutBlacklist SQLite error: code 13 errno 112: database or disk is full sql: CREATE TABLE IF NOT EXISTS previews_v1 (host_name VARCHAR NOT NULL, time INTEGER NOT NULL, opt_out INTEGER NOT
NULL, type INTEGER NOT NULL, PRIMARY KEY(host_name, time DESC, opt_out, type))
[19980:18944:0316/183240.121:ERROR:database.cc(1777)] OptOutBlacklist SQLite error: code 1 errno 112: no such table: enabled_previews_v1 sql: SELECT type, version FROM enabled_previews_v1
[19980:18944:0316/183240.132:ERROR:database.cc(1777)] OptOutBlacklist SQLite error: code 1 errno 112: no such table: enabled_previews_v1 sql: INSERT INTO enabled_previews_v1 (type, version) VALUES (?, ?)
[19980:18944:0316/183240.180:ERROR:database.cc(1777)] OptOutBlacklist SQLite error: code 1 errno 112: no such table: previews_v1 sql: SELECT host_name, time, opt_out, type FROM previews_v1 ORDER BY host_name, time DESC
[15816:23948:0316/183240.237:ERROR:entry_impl.cc(999)] Failed to save user data
[15816:23948:0316/183240.259:ERROR:entry_impl.cc(999)] Failed to save user data
[15816:23948:0316/183240.340:ERROR:entry_impl.cc(999)] Failed to save user data
[15816:23948:0316/183240.359:ERROR:entry_impl.cc(999)] Failed to save user data
[19980:18944:0316/183240.607:ERROR:database.cc(1777)] Shortcuts SQLite error: code 13 errno 112: database or disk is full sql: COMMIT
[15816:23948:0316/183240.677:ERROR:entry_impl.cc(999)] Failed to save user data
[15816:23948:0316/183240.755:ERROR:entry_impl.cc(999)] Failed to save user data
[15816:23948:0316/183240.765:ERROR:entry_impl.cc(999)] Failed to save user data
[19980:18944:0316/183240.825:ERROR:database.cc(1777)] Shortcuts SQLite error: code 1 errno 112: no such table: omni_box_shortcuts sql: SELECT id, text, fill_into_edit, url, document_type, contents, contents_class, description, description_class, transition, type, keyword, last_access_time, number_of_hits FROM omni_box_shortcuts
[19980:19372:0316/183241.460:ERROR:database.cc(1777)] TopSites SQLite error: code 13 errno 0: database or disk is full sql: INSERT OR REPLACE INTO top_sites (url, url_rank, title) VALUES (?, ?, ?)
[19980:19372:0316/183241.468:ERROR:database.cc(1777)] TopSites SQLite error: code 13 errno 0: database or disk is full sql: INSERT OR REPLACE INTO top_sites (url, url_rank, title) VALUES (?, ?, ?)
[19980:19372:0316/183241.479:ERROR:database.cc(1777)] TopSites SQLite error: code 13 errno 0: database or disk is full sql: UPDATE top_sites SET url_rank = ? WHERE url == ?
[19980:19372:0316/183241.479:ERROR:database.cc(1777)] TopSites SQLite error: code 1 errno 0: cannot commit - no transaction is active sql: COMMIT
[19980:18944:0316/183241.550:ERROR:database.cc(1777)] Predictor SQLite error: code 13 errno 112: database or disk is full sql: COMMIT
我基本上用额外的 js 修复了它
driver.execute_script("""(function(console){
console.save = function(data, filename){
if(!data) {
console.error('Console.save: No data')
return;
}
if(!filename) filename = 'console.json'
if(typeof data === "object"){
data = JSON.stringify(data, undefined, 4)
}
var blob = new Blob([data], {type: 'text/json'}),
e = document.createEvent('MouseEvents'),
a = document.createElement('a')
a.download = filename
a.href = window.URL.createObjectURL(blob)
a.dataset.downloadurl = ['text/json', a.download, a.href].join(':')
e.initMouseEvent('click', true, false, window, 0, 0, 0, 0, 0, false, false, false, false, 0, null)
a.dispatchEvent(e)
}
})(console)""")
driver.execute_script('values = [];Highcharts.charts[0].series[0].data.forEach(function(d){ values.push(d.y) });')
driver.execute_script('console.save(values, add);')
这会将数组保存到 json
我认为是XYProblem。
你问如何从 console.log() 获取输出，但似乎真正的问题是如何从 Highcharts 获取数据并在 Python 中使用 - 而这不需要 console.log().
你可以在JavaScript中使用return values在Python中得到它。
它甚至不需要 JavaScript 中的 JSON.stringify() 和 Python 中的 json.loads()。
data = driver.execute_script('''
values = [];
Highcharts.charts[0].series[0].data.forEach((d) => values.push(d.y));
return values;
''')
print(data)
编辑：
使用 .map() 而不是 forEarch() 你可以将 JavaScript 减少到一行：
return Highcharts.charts[0].series[0].data.map(d => d.y);
https://www.highcharts.com/demo/line-basic
的最小工作示例from selenium import webdriver
#from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.firefox import GeckoDriverManager
import time
url = 'https://www.highcharts.com/demo/line-basic'
#driver = webdriver.Chrome(executable_path=ChromeDriverManager().install())
driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
driver.get(url)
time.sleep(5)
data = driver.execute_script('''
values = [];
Highcharts.charts[0].series[0].data.forEach((d) => values.push(d.y));
return values;
''')
print(type(data), data)
# or shorter
data = driver.execute_script('return Highcharts.charts[0].series[0].data.map(d => d.y);')
print(type(data), data)
结果：
<class 'list'> [43934, 52503, 57177, 69658, 97031, 119931, 137133, 154175]
编辑：
例如，它可用于将带有时间戳（以毫秒为单位）的 x 转换为 MM/DD/YY 或 YYYY-MM-DD
import datetime
data = [1647212400000, 1647298800000, 1647385200000, 1647471600000]
new_data = []
for x in data:
dt = datetime.datetime.fromtimestamp(x//1000)
new_data.append(d.strftime('%D'))
print(d.strftime('%D'), '|', d.strftime('%Y-%m-%d'))
print('new_data:', new_data)
结果：
03/17/22 | 2022-03-17
03/17/22 | 2022-03-17
03/17/22 | 2022-03-17
03/17/22 | 2022-03-17
new_data: ['03/17/22', '03/17/22', '03/17/22', '03/17/22']
如果您将数据保存在 pandas.DataFrame 中，那么您可以使用 pd.to_datetime() 和 .dt.strftime('%D') 但是为了计算，最好将其保存为 datetime
import pandas as pd
data = [1647212400000, 1647298800000, 1647385200000, 1647471600000]
df = pd.DataFrame({'date': data})
print(df)
df['date'] = pd.to_datetime(df['date'], unit='ms')
df['date'] = df['date'].dt.strftime('%D')
print(df)
结果：
date
0 1647212400000
1 1647298800000
2 1647385200000
3 1647471600000
date
0 03/13/22
1 03/14/22
2 03/15/22
3 03/16/22