Invoke-Webrequest 不同于浏览器
Invoke-Webrequest different from browser
我有一个脚本可以将 html table 解析为 powershell table。
特定网站更新后脚本损坏，现在当我尝试解析 table 时，我得到错误信息（当我从 Internet Explorer 11 打开网站时也是如此）。
这是代码：
Get-HtmlTable
Function Get-HTMLTables{
<#
.SYNOPSIS Gets all the text in HTML tables on a webpage and returns it as an object of arrays
.DESCRIPTION
This function treats
1. table cells as properties of row objects
2. Rows as custom objects with cells as properties
3. Tables as arrays of rows
4. Pages as custom objects with numbered tables as properties.
Returns
-Custom object with numbered properties
-Each numbered property represents a table number from the top of the page down
-Each numbered property contains an array of row custom objects
-Each object in the array has properties representing each cell in the row.
.PARAMETER URL
The URL to look for HTML tables in
.PARAMETER FirstRowHeader
Whether or not to treat the first row of the table as a header
.EXAMPLE
$objPage = Get-HTMLTables -URL 'https://support.microsoft.com/en-us/lifecycle/search?sort=PN&alpha=windows%2010&Filter=FilterNO' -FirstRowHeader $true
.EXAMPLE
$objPage.21 | select -first 1
Extended Support End Date : 10/14/2025
Lifecycle Start Date : 7/29/2015
Products Released : Visual Studio Express 2015 for Windows 10
Mainstream Support End Date : 10/13/2020
Service Pack Support End Date :
Notes :
.EXAMPLE
$objPage.21 | Where{$_.('Products Released') -match 'Education'}
Extended Support End Date : 10/14/2025
Lifecycle Start Date : 7/29/2015
Products Released : Windows 10 Education, released in July 2015
Mainstream Support End Date : 10/13/2020
Service Pack Support End Date :
Notes : • Updates are cumulative, with each update built upon all of the updates that preceded it. A device needs to
install the latest update to remain supported.
• Updates may include new features, fixes (security and/or non-security), or a combination of both. Not all
features in an update will work on all devices.
• A device may not be able to receive updates if the device hardware is incompatible, lacking current
drivers, or otherwise outside of the Original Equipment Manufacturer’s (“OEM”) support period.
• Update availability may vary, for example by country, region, network connectivity, mobile operator (e.g.,
for cellular-capable devices), or hardware capabilities (including, e.g., free disk space).
#>
Param(
[uri]$URL = 'https://www.enterprisedb.com/downloads/postgres-postgresql-downloads',
[boolean]$firstRowHeader = $false
)
$AllProtocols = [System.Net.SecurityProtocolType]'Ssl3,Tls,Tls11,Tls12'
[System.Net.ServicePointManager]::SecurityProtocol = $AllProtocols
#Useragent
$userAgent = [Microsoft.PowerShell.Commands.PSUserAgent]::Safari
#Get the webpage
$page = Invoke-WebRequest $URL -UserAgent 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36'
#Filter out only the tables
$tables = $page.ParsedHtml.body.getElementsByTagName('Table')
#Get only the tables that have cells
$tableswithcells = $tables | Where{$_.cells}
$hashPage = @{}
$tablecount = 0
#ForEach table
ForEach($table in $tableswithcells){
$arrTable = @()
$rownum = 0
$arrTableHeader = @()
#Get all the rows in the tables
ForEach($row in $table.rows){
#Treat the first row as a header
if($rownum -eq 0 -and $firstRowHeader){
ForEach($cell in $row.cells){
$arrTableHeader += $cell.InnerText.Trim()
}
#If not the first row, but using headers, store the value by header name
}elseIf($firstRowHeader){
$cellnum = 0
$hashRow = @{}
ForEach($cell in $row.cells){
$strHeader = $arrTableHeader[$cellNum]
If($strHeader){
$hashRow.Add($strHeader,(($cell.innerhtml -replace "(?s)^.*$href=`"", '') -replace "`">Download</a>", ""))
#$hashRow.Add($strHeader,$cell.innertext)
}else{
#If the header is null store it by cell number instead
$hashRow.Add($cellnum,$cell.innertext)
}
$cellnum++
}
#Save the row as a custom ps object
$objRow = New-object -TypeName PSCustomObject -Property $hashRow
$arrTable += $objRow
#if not the first row and not using headers, store the value by cell index
}else{
$cellnum = 0
$hashRow = @{}
ForEach($cell in $row.cells){
$hashRow.Add($cellnum,$cell.innertext)
$cellnum++
}
#Store the row as a custom object
$objRow = New-object -TypeName PSCustomObject -Property $hashRow
#Add the row to the array of rows
$arrTable += $objRow
}
$rownum++
}
#Add the tables to the hashtable of tables
$hashPage.Add($tablecount,$arrTable)
$tablecount++
}
$objPage = New-object -TypeName PSCustomObject -Property $hashPage
Return $objPage
}
当我执行以下行时：
$test = Get-HTMLTables -firstRowHeader $false -url 'https://www.enterprisedb.com/downloads/postgres-postgresql-downloads'
$test.0
结果如下：
5 : Windows x86-32
4 : Windows x86-64
3 : Mac OS X
2 : Linux x86-32
1 : Linux x86-64
0 : PostgreSQL Version
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 14.1
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 13.5
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 12.9
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 11.14
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 10.19
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 9.6.24*
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 9.5.25*
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 9.4.26*
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 9.3.25*
我要抓取的网站：
https://www.enterprisedb.com/downloads/postgres-postgresql-downloads
如您所见，有适用于 Win64 的可用版本。
使用 Internet Explorer 11 打开此网站时也会发生同样的情况，尝试更改用户代理但问题仍然存在。
正如 @Mathias: the required (Json 评论的那样）数据在 (__NEXT_DATA__) 脚本中。
要提取该信息，您需要执行以下操作：
$Page = Invoke-WebRequest 'https://www.enterprisedb.com/downloads/postgres-postgresql-downloads'
$Unicode = [System.Text.Encoding]::Unicode.GetBytes($Page.Content)
$Document = New-Object -Com 'HTMLFile'
if ($Document.IHTMLDocument2_Write) { $Document.IHTMLDocument2_Write($Unicode) } else { $Document.write($Unicode) }
$Document.Close()
$Data = $Document.getElementById('__NEXT_DATA__').innerHTML |ConvertFrom-Json
$Data |ConvertTo-Json -Depth 10
产量：
{
"props": {
"pageProps": {
"postgreSQLDownloads": [
{
"version": "14.1",
"products": [
{
"title": "PostgreSQL Download",
"field_installer_version": "14",
"field_sub_version": "1",
"field_os": "Mac OS X",
"field_installation_method": "Interactive Installer",
"field_supported_os": "MacOSX 10.12+",
"uuid": "f027c016-7c5b-43fd-beb7-59ee43135607",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001NhszQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
},
{
"title": "PostgreSQL Download",
"field_installer_version": "14",
"field_sub_version": "1",
"field_os": "Windows x86-64",
"field_installation_method": "Interactive Installer",
"field_supported_os": "Windows Server 2016, Windows Server 2019",
"uuid": "db55e32d-e9f0-4d7c-9aef-b17d01210704",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001NhszQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
}
]
},
{
"version": "13.5",
"products": [
{
"title": "PostgreSQL Download",
"field_installer_version": "13",
"field_sub_version": "5",
"field_os": "Linux x86-64",
"field_installation_method": "RPM",
"field_supported_os": "CentOS 8 / 7, OL 8 / 7, RHEL 8 / 7, SLES 12",
"uuid": "514cbe6a-089c-4cb9-a6b1-ad9d6375842f",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001BfmaQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
},
{
"title": "PostgreSQL Download",
"field_installer_version": "13",
"field_sub_version": "5",
"field_os": "Linux x86-64",
"field_installation_method": "DEB",
"field_supported_os": "Debian 10, Debian 9.6.x, Ubuntu 18.04, Ubuntu 20.04",
"uuid": "e3c7350d-e18c-4eae-8817-c61cb83e09b9",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001BfmaQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
},
{
"title": "PostgreSQL Download",
"field_installer_version": "13",
"field_sub_version": "5",
"field_os": "Mac OS X",
"field_installation_method": "Interactive Installer",
"field_supported_os": "MacOSX 10.12+",
"uuid": "76003af2-5f27-4b85-9804-c3b2237debf5",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001BfmaQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
},
{
"title": "PostgreSQL Download",
"field_installer_version": "13",
"field_sub_version": "5",
"field_os": "Windows x86-64",
"field_installation_method": "Interactive Installer",
"field_supported_os": "Windows Server 2012 R2, Windows Server 2016, Windows Server 2019",
"uuid": "7c756686-90b4-4909-89ed-043e0705a76e",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001BfmaQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
}
]
}
// ...
]
},
"__N_SSP": true
},
"page": "/downloads/postgres-postgresql-downloads",
"query": {},
"buildId": "u3IIRJVKDxUWWWroinlsJ",
"runtimeConfig": {},
"isFallback": false,
"gssp": true,
"scriptLoader": []
}
和
$Data.props.pageProps.postgreSQLDownloads.products |Format-Table
产量：
title field_installer_version field_sub_version field_os field_installation_method field_supported_os uuid field_show_this_version field_campaign_id field_product_category
----- ----------------------- ----------------- -------- ------------------------- ------------------ ---- ----------------------- ----------------- ----------------------
PostgreSQL Download 14 1 Mac OS X Interactive Installer MacOSX 10.12+ f027c016-7c5b-43fd-beb7-59ee43135607 1 7012J000001NhszQAC PostgreSQL
PostgreSQL Download 14 1 Windows x86-64 Interactive Installer Windows Server 2016, Windows Server 2019 db55e32d-e9f0-4d7c-9aef-b17d01210704 1 7012J000001NhszQAC PostgreSQL
PostgreSQL Download 13 5 Linux x86-64 RPM CentOS 8 / 7, OL 8 / 7, RHEL 8 / 7, SLES 12 514cbe6a-089c-4cb9-a6b1-ad9d6375842f 1 7012J000001BfmaQAC PostgreSQL
PostgreSQL Download 13 5 Linux x86-64 DEB Debian 10, Debian 9.6.x, Ubuntu 18.04, Ubuntu 20.04 e3c7350d-e18c-4eae-8817-c61cb83e09b9 1 7012J000001BfmaQAC PostgreSQL
PostgreSQL Download 13 5 Mac OS X Interactive Installer MacOSX 10.12+ 76003af2-5f27-4b85-9804-c3b2237debf5 1 7012J000001BfmaQAC PostgreSQL
PostgreSQL Download 13 5 Windows x86-64 Interactive Installer Windows Server 2012 R2, Windows Server 2016, Windows Server 2019 7c756686-90b4-4909-89ed-043e0705a76e 1 7012J000001BfmaQAC PostgreSQL
PostgreSQL Download 12 9 Linux x86-64 RPM CentOS 8 / 7, OL 8 / 7, RHEL 8 / 7, SLES 12 9a8d19d8-980c-4475-af6c-31aa964d2a30 1 7012J000001F5IIQA0 PostgreSQL
PostgreSQL Download 12 9 Linux x86-64 DEB Debian 9.6.x, Ubuntu 18.04 0753b404-99a7-4c0d-867a-8e9534a566c7 1 7012J000001F5IIQA0 PostgreSQL
PostgreSQL Download 12 9 Mac OS X Interactive Installer MacOSX 10.12+ 942ce37d-8817-42be-8466-51f653728eda 1 7012J000001F5IIQA0 PostgreSQL
PostgreSQL Download 12 9 Windows x86-64 Interactive Installer Windows Server 2012 R2, Windows Server 2016, Windows Server 2019 03b13357-9281-4985-baea-17b72a0febc3 1 7012J000001F5IIQA0 PostgreSQL
PostgreSQL Download 11 14 Linux x86-64 RPM ababa720-d9ac-46fa-88a0-161c27bdc4cf 1 70138000001U9DfAAK PostgreSQL
PostgreSQL Download 11 14 Linux x86-64 & 32 DEB befdfd14-abe0-4614-b22b-0b4a577dfd7f 1 70138000001U9DfAAK PostgreSQL
PostgreSQL Download 11 14 Mac OS X Interactive Installer MacOSX 10.12+ dcf30630-6453-4c50-9de1-add7a8bf431c 1 70138000001U9DfAAK PostgreSQL
PostgreSQL Download 11 14 Windows x86-64 Interactive Installer Windows Server 2012 R2, Windows Server 2016, Windows Server 2019 995d0688-d8cf-48b8-8afe-13c777b0ef8e 1 70138000001U9DfAAK PostgreSQL
PostgreSQL Download 10 19 Linux x86-32 Interactive Installer SLES 12, Ubuntu 14.04 f6607ede-6698-4116-bf56-bd25284203aa 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Linux x86-64 Interactive Installer CentOS 7, Debian 8 / 7, OL 7, RHEL 7, Ubuntu 14.04 beb6b65f-6aeb-466f-8b9f-c27a7e327ccf 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Linux x86-64 RPM 03f51b70-24ee-4c75-88a7-de2e488c59da 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Linux x86-64 & 32 DEB a92ccaab-7c3f-4a5b-9a5e-e632b79b33d7 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Mac OS X Interactive Installer MacOSX 10.12+ c1b50eb7-9ad6-4e6c-a094-9a7f70ee86f9 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Windows x86-32 Interactive Installer Windows 7 / 8 / 10 e9a60157-955a-4f24-a738-07cff51ccb43 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Windows x86-64 Windows 2012 R2 / R1, Windows 7 / 8 / 10, Windows Server 2016 ea5c8104-3940-4ed1-b427-81cf19781581 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 9 6.24 Linux x86-32 Interactive Installer SLES 12, Ubuntu 14.04 a9b88ba3-839b-4883-b2c5-378d4a7ee3da 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Linux x86-64 Interactive Installer CentOS 7, Debian 8 / 7, OL 7, RHEL 7, SLES 12, Ubuntu 14.04 c539f198-0912-46a9-b4e6-545166574893 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Linux x86-64 RPM ceb7a789-4a0c-4c17-8d48-abb1cdfd9cff 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Linux x86-64 & 32 DEB b976efbe-41d1-4d3f-9b14-95b7b03e651e 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Mac OS X Interactive Installer MacOSX 10.12+ b40e8d78-4a3b-458a-85f5-f0b7e0bbfa31 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Windows x86-32 Interactive Installer Windows 2008 R1, Windows 7 / 8 / 10 5011320c-34df-4c8f-9b6f-21e53c4a4930 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Windows x86-64 Interactive Installer Windows 2008 R2, Windows 2012 R2 / R1, Windows 7 / 8 / 10 91f8264f-8a2d-434a-9559-4dfa9b3b80b1 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 5.25 Linux x86-32 Interactive Installer SLES 12, Ubuntu 14.04 25ec7179-be39-4e0f-a429-bc140ccfab00 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Linux x86-64 Interactive Installer Debian 7, SLES 12, Ubuntu 14.04 147a6582-849c-44d1-8a51-e90fcece30a6 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Linux x86-64 RPM 5310449a-13b3-4ca1-922c-f0baef05c3ab 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Linux x86-64 & 32 DEB 654437d3-3c63-449c-bd22-1deda40dad16 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Mac OS X Interactive Installer MacOSX 10.10, MacOSX 10.8, MacOSX 10.9 3a9e2080-a11f-4230-8974-78c2ef7234d9 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Windows x86-32 Interactive Installer Windows 2008 R1, Windows 7 / 8 / 10 aa5aa60c-2030-4e06-a6ba-06ba0eb79383 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Windows x86-64 Interactive Installer Windows 2008 R2, Windows 2012 R2 / R1 7869632c-134f-4284-8fba-feb67010b372 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 4.26 Linux x86-32 Interactive Installer ed2f791f-ca94-4872-ac52-d3221e1c1950 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 4.26 Linux x86-64 Interactive Installer 5273bfc5-9382-4a55-ab9a-d42c86296dac 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 4.26 Mac OS X Interactive Installer 58b175c3-0c86-425a-a3fa-5bbdb10abf08 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 4.26 Windows x86-32 Interactive Installer 33e923ed-0d9e-40b1-8972-96c245d239e4 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 4.26 Windows x86-64 Interactive Installer cba59404-da4d-4009-8edb-6655b72464dd 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 3.25 Linux x86-32 Interactive Installer 0e139bf2-f0da-4ba9-bb40-9c0cadb4087e 0 70150000000rxXPAAY PostgreSQL
PostgreSQL Download 9 3.25 Linux x86-64 Interactive Installer 566673e2-7d5e-46aa-82dc-7400d296400e 0 70150000000rxXPAAY PostgreSQL
PostgreSQL Download 9 3.25 Mac OS X Interactive Installer 72c6d5f4-68bb-4080-a8cf-769f5fc60f41 0 70150000000rxXPAAY PostgreSQL
PostgreSQL Download 9 3.25 Windows x86-32 Interactive Installer 46c086f2-944c-49a2-bac3-204ccf14ac17 0 70150000000rxXPAAY PostgreSQL
PostgreSQL Download 9 3.25 Windows x86-64 Interactive Installer dbb5a7ac-4ecb-4a11-be2b-d56e8908b2a4 0 70150000000rxXPAAY PostgreSQL
我有一个脚本可以将 html table 解析为 powershell table。 特定网站更新后脚本损坏，现在当我尝试解析 table 时，我得到错误信息（当我从 Internet Explorer 11 打开网站时也是如此）。 这是代码：
Get-HtmlTable
Function Get-HTMLTables{
<#
.SYNOPSIS Gets all the text in HTML tables on a webpage and returns it as an object of arrays
.DESCRIPTION
This function treats
1. table cells as properties of row objects
2. Rows as custom objects with cells as properties
3. Tables as arrays of rows
4. Pages as custom objects with numbered tables as properties.
Returns
-Custom object with numbered properties
-Each numbered property represents a table number from the top of the page down
-Each numbered property contains an array of row custom objects
-Each object in the array has properties representing each cell in the row.
.PARAMETER URL
The URL to look for HTML tables in
.PARAMETER FirstRowHeader
Whether or not to treat the first row of the table as a header
.EXAMPLE
$objPage = Get-HTMLTables -URL 'https://support.microsoft.com/en-us/lifecycle/search?sort=PN&alpha=windows%2010&Filter=FilterNO' -FirstRowHeader $true
.EXAMPLE
$objPage.21 | select -first 1
Extended Support End Date : 10/14/2025
Lifecycle Start Date : 7/29/2015
Products Released : Visual Studio Express 2015 for Windows 10
Mainstream Support End Date : 10/13/2020
Service Pack Support End Date :
Notes :
.EXAMPLE
$objPage.21 | Where{$_.('Products Released') -match 'Education'}
Extended Support End Date : 10/14/2025
Lifecycle Start Date : 7/29/2015
Products Released : Windows 10 Education, released in July 2015
Mainstream Support End Date : 10/13/2020
Service Pack Support End Date :
Notes : • Updates are cumulative, with each update built upon all of the updates that preceded it. A device needs to
install the latest update to remain supported.
• Updates may include new features, fixes (security and/or non-security), or a combination of both. Not all
features in an update will work on all devices.
• A device may not be able to receive updates if the device hardware is incompatible, lacking current
drivers, or otherwise outside of the Original Equipment Manufacturer’s (“OEM”) support period.
• Update availability may vary, for example by country, region, network connectivity, mobile operator (e.g.,
for cellular-capable devices), or hardware capabilities (including, e.g., free disk space).
#>
Param(
[uri]$URL = 'https://www.enterprisedb.com/downloads/postgres-postgresql-downloads',
[boolean]$firstRowHeader = $false
)
$AllProtocols = [System.Net.SecurityProtocolType]'Ssl3,Tls,Tls11,Tls12'
[System.Net.ServicePointManager]::SecurityProtocol = $AllProtocols
#Useragent
$userAgent = [Microsoft.PowerShell.Commands.PSUserAgent]::Safari
#Get the webpage
$page = Invoke-WebRequest $URL -UserAgent 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36'
#Filter out only the tables
$tables = $page.ParsedHtml.body.getElementsByTagName('Table')
#Get only the tables that have cells
$tableswithcells = $tables | Where{$_.cells}
$hashPage = @{}
$tablecount = 0
#ForEach table
ForEach($table in $tableswithcells){
$arrTable = @()
$rownum = 0
$arrTableHeader = @()
#Get all the rows in the tables
ForEach($row in $table.rows){
#Treat the first row as a header
if($rownum -eq 0 -and $firstRowHeader){
ForEach($cell in $row.cells){
$arrTableHeader += $cell.InnerText.Trim()
}
#If not the first row, but using headers, store the value by header name
}elseIf($firstRowHeader){
$cellnum = 0
$hashRow = @{}
ForEach($cell in $row.cells){
$strHeader = $arrTableHeader[$cellNum]
If($strHeader){
$hashRow.Add($strHeader,(($cell.innerhtml -replace "(?s)^.*$href=`"", '') -replace "`">Download</a>", ""))
#$hashRow.Add($strHeader,$cell.innertext)
}else{
#If the header is null store it by cell number instead
$hashRow.Add($cellnum,$cell.innertext)
}
$cellnum++
}
#Save the row as a custom ps object
$objRow = New-object -TypeName PSCustomObject -Property $hashRow
$arrTable += $objRow
#if not the first row and not using headers, store the value by cell index
}else{
$cellnum = 0
$hashRow = @{}
ForEach($cell in $row.cells){
$hashRow.Add($cellnum,$cell.innertext)
$cellnum++
}
#Store the row as a custom object
$objRow = New-object -TypeName PSCustomObject -Property $hashRow
#Add the row to the array of rows
$arrTable += $objRow
}
$rownum++
}
#Add the tables to the hashtable of tables
$hashPage.Add($tablecount,$arrTable)
$tablecount++
}
$objPage = New-object -TypeName PSCustomObject -Property $hashPage
Return $objPage
}
当我执行以下行时：
$test = Get-HTMLTables -firstRowHeader $false -url 'https://www.enterprisedb.com/downloads/postgres-postgresql-downloads'
$test.0
结果如下：
5 : Windows x86-32
4 : Windows x86-64
3 : Mac OS X
2 : Linux x86-32
1 : Linux x86-64
0 : PostgreSQL Version
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 14.1
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 13.5
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 12.9
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 11.14
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 10.19
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 9.6.24*
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 9.5.25*
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 9.4.26*
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 9.3.25*
我要抓取的网站： https://www.enterprisedb.com/downloads/postgres-postgresql-downloads 如您所见，有适用于 Win64 的可用版本。
使用 Internet Explorer 11 打开此网站时也会发生同样的情况，尝试更改用户代理但问题仍然存在。
正如 @Mathias: the required (Json 评论的那样）数据在 (__NEXT_DATA__) 脚本中。
要提取该信息，您需要执行以下操作：
$Page = Invoke-WebRequest 'https://www.enterprisedb.com/downloads/postgres-postgresql-downloads'
$Unicode = [System.Text.Encoding]::Unicode.GetBytes($Page.Content)
$Document = New-Object -Com 'HTMLFile'
if ($Document.IHTMLDocument2_Write) { $Document.IHTMLDocument2_Write($Unicode) } else { $Document.write($Unicode) }
$Document.Close()
$Data = $Document.getElementById('__NEXT_DATA__').innerHTML |ConvertFrom-Json
$Data |ConvertTo-Json -Depth 10
产量：
{
"props": {
"pageProps": {
"postgreSQLDownloads": [
{
"version": "14.1",
"products": [
{
"title": "PostgreSQL Download",
"field_installer_version": "14",
"field_sub_version": "1",
"field_os": "Mac OS X",
"field_installation_method": "Interactive Installer",
"field_supported_os": "MacOSX 10.12+",
"uuid": "f027c016-7c5b-43fd-beb7-59ee43135607",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001NhszQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
},
{
"title": "PostgreSQL Download",
"field_installer_version": "14",
"field_sub_version": "1",
"field_os": "Windows x86-64",
"field_installation_method": "Interactive Installer",
"field_supported_os": "Windows Server 2016, Windows Server 2019",
"uuid": "db55e32d-e9f0-4d7c-9aef-b17d01210704",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001NhszQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
}
]
},
{
"version": "13.5",
"products": [
{
"title": "PostgreSQL Download",
"field_installer_version": "13",
"field_sub_version": "5",
"field_os": "Linux x86-64",
"field_installation_method": "RPM",
"field_supported_os": "CentOS 8 / 7, OL 8 / 7, RHEL 8 / 7, SLES 12",
"uuid": "514cbe6a-089c-4cb9-a6b1-ad9d6375842f",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001BfmaQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
},
{
"title": "PostgreSQL Download",
"field_installer_version": "13",
"field_sub_version": "5",
"field_os": "Linux x86-64",
"field_installation_method": "DEB",
"field_supported_os": "Debian 10, Debian 9.6.x, Ubuntu 18.04, Ubuntu 20.04",
"uuid": "e3c7350d-e18c-4eae-8817-c61cb83e09b9",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001BfmaQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
},
{
"title": "PostgreSQL Download",
"field_installer_version": "13",
"field_sub_version": "5",
"field_os": "Mac OS X",
"field_installation_method": "Interactive Installer",
"field_supported_os": "MacOSX 10.12+",
"uuid": "76003af2-5f27-4b85-9804-c3b2237debf5",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001BfmaQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
},
{
"title": "PostgreSQL Download",
"field_installer_version": "13",
"field_sub_version": "5",
"field_os": "Windows x86-64",
"field_installation_method": "Interactive Installer",
"field_supported_os": "Windows Server 2012 R2, Windows Server 2016, Windows Server 2019",
"uuid": "7c756686-90b4-4909-89ed-043e0705a76e",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001BfmaQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
}
]
}
// ...
]
},
"__N_SSP": true
},
"page": "/downloads/postgres-postgresql-downloads",
"query": {},
"buildId": "u3IIRJVKDxUWWWroinlsJ",
"runtimeConfig": {},
"isFallback": false,
"gssp": true,
"scriptLoader": []
}
和
$Data.props.pageProps.postgreSQLDownloads.products |Format-Table
产量：
title field_installer_version field_sub_version field_os field_installation_method field_supported_os uuid field_show_this_version field_campaign_id field_product_category
----- ----------------------- ----------------- -------- ------------------------- ------------------ ---- ----------------------- ----------------- ----------------------
PostgreSQL Download 14 1 Mac OS X Interactive Installer MacOSX 10.12+ f027c016-7c5b-43fd-beb7-59ee43135607 1 7012J000001NhszQAC PostgreSQL
PostgreSQL Download 14 1 Windows x86-64 Interactive Installer Windows Server 2016, Windows Server 2019 db55e32d-e9f0-4d7c-9aef-b17d01210704 1 7012J000001NhszQAC PostgreSQL
PostgreSQL Download 13 5 Linux x86-64 RPM CentOS 8 / 7, OL 8 / 7, RHEL 8 / 7, SLES 12 514cbe6a-089c-4cb9-a6b1-ad9d6375842f 1 7012J000001BfmaQAC PostgreSQL
PostgreSQL Download 13 5 Linux x86-64 DEB Debian 10, Debian 9.6.x, Ubuntu 18.04, Ubuntu 20.04 e3c7350d-e18c-4eae-8817-c61cb83e09b9 1 7012J000001BfmaQAC PostgreSQL
PostgreSQL Download 13 5 Mac OS X Interactive Installer MacOSX 10.12+ 76003af2-5f27-4b85-9804-c3b2237debf5 1 7012J000001BfmaQAC PostgreSQL
PostgreSQL Download 13 5 Windows x86-64 Interactive Installer Windows Server 2012 R2, Windows Server 2016, Windows Server 2019 7c756686-90b4-4909-89ed-043e0705a76e 1 7012J000001BfmaQAC PostgreSQL
PostgreSQL Download 12 9 Linux x86-64 RPM CentOS 8 / 7, OL 8 / 7, RHEL 8 / 7, SLES 12 9a8d19d8-980c-4475-af6c-31aa964d2a30 1 7012J000001F5IIQA0 PostgreSQL
PostgreSQL Download 12 9 Linux x86-64 DEB Debian 9.6.x, Ubuntu 18.04 0753b404-99a7-4c0d-867a-8e9534a566c7 1 7012J000001F5IIQA0 PostgreSQL
PostgreSQL Download 12 9 Mac OS X Interactive Installer MacOSX 10.12+ 942ce37d-8817-42be-8466-51f653728eda 1 7012J000001F5IIQA0 PostgreSQL
PostgreSQL Download 12 9 Windows x86-64 Interactive Installer Windows Server 2012 R2, Windows Server 2016, Windows Server 2019 03b13357-9281-4985-baea-17b72a0febc3 1 7012J000001F5IIQA0 PostgreSQL
PostgreSQL Download 11 14 Linux x86-64 RPM ababa720-d9ac-46fa-88a0-161c27bdc4cf 1 70138000001U9DfAAK PostgreSQL
PostgreSQL Download 11 14 Linux x86-64 & 32 DEB befdfd14-abe0-4614-b22b-0b4a577dfd7f 1 70138000001U9DfAAK PostgreSQL
PostgreSQL Download 11 14 Mac OS X Interactive Installer MacOSX 10.12+ dcf30630-6453-4c50-9de1-add7a8bf431c 1 70138000001U9DfAAK PostgreSQL
PostgreSQL Download 11 14 Windows x86-64 Interactive Installer Windows Server 2012 R2, Windows Server 2016, Windows Server 2019 995d0688-d8cf-48b8-8afe-13c777b0ef8e 1 70138000001U9DfAAK PostgreSQL
PostgreSQL Download 10 19 Linux x86-32 Interactive Installer SLES 12, Ubuntu 14.04 f6607ede-6698-4116-bf56-bd25284203aa 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Linux x86-64 Interactive Installer CentOS 7, Debian 8 / 7, OL 7, RHEL 7, Ubuntu 14.04 beb6b65f-6aeb-466f-8b9f-c27a7e327ccf 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Linux x86-64 RPM 03f51b70-24ee-4c75-88a7-de2e488c59da 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Linux x86-64 & 32 DEB a92ccaab-7c3f-4a5b-9a5e-e632b79b33d7 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Mac OS X Interactive Installer MacOSX 10.12+ c1b50eb7-9ad6-4e6c-a094-9a7f70ee86f9 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Windows x86-32 Interactive Installer Windows 7 / 8 / 10 e9a60157-955a-4f24-a738-07cff51ccb43 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Windows x86-64 Windows 2012 R2 / R1, Windows 7 / 8 / 10, Windows Server 2016 ea5c8104-3940-4ed1-b427-81cf19781581 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 9 6.24 Linux x86-32 Interactive Installer SLES 12, Ubuntu 14.04 a9b88ba3-839b-4883-b2c5-378d4a7ee3da 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Linux x86-64 Interactive Installer CentOS 7, Debian 8 / 7, OL 7, RHEL 7, SLES 12, Ubuntu 14.04 c539f198-0912-46a9-b4e6-545166574893 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Linux x86-64 RPM ceb7a789-4a0c-4c17-8d48-abb1cdfd9cff 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Linux x86-64 & 32 DEB b976efbe-41d1-4d3f-9b14-95b7b03e651e 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Mac OS X Interactive Installer MacOSX 10.12+ b40e8d78-4a3b-458a-85f5-f0b7e0bbfa31 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Windows x86-32 Interactive Installer Windows 2008 R1, Windows 7 / 8 / 10 5011320c-34df-4c8f-9b6f-21e53c4a4930 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Windows x86-64 Interactive Installer Windows 2008 R2, Windows 2012 R2 / R1, Windows 7 / 8 / 10 91f8264f-8a2d-434a-9559-4dfa9b3b80b1 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 5.25 Linux x86-32 Interactive Installer SLES 12, Ubuntu 14.04 25ec7179-be39-4e0f-a429-bc140ccfab00 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Linux x86-64 Interactive Installer Debian 7, SLES 12, Ubuntu 14.04 147a6582-849c-44d1-8a51-e90fcece30a6 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Linux x86-64 RPM 5310449a-13b3-4ca1-922c-f0baef05c3ab 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Linux x86-64 & 32 DEB 654437d3-3c63-449c-bd22-1deda40dad16 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Mac OS X Interactive Installer MacOSX 10.10, MacOSX 10.8, MacOSX 10.9 3a9e2080-a11f-4230-8974-78c2ef7234d9 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Windows x86-32 Interactive Installer Windows 2008 R1, Windows 7 / 8 / 10 aa5aa60c-2030-4e06-a6ba-06ba0eb79383 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Windows x86-64 Interactive Installer Windows 2008 R2, Windows 2012 R2 / R1 7869632c-134f-4284-8fba-feb67010b372 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 4.26 Linux x86-32 Interactive Installer ed2f791f-ca94-4872-ac52-d3221e1c1950 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 4.26 Linux x86-64 Interactive Installer 5273bfc5-9382-4a55-ab9a-d42c86296dac 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 4.26 Mac OS X Interactive Installer 58b175c3-0c86-425a-a3fa-5bbdb10abf08 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 4.26 Windows x86-32 Interactive Installer 33e923ed-0d9e-40b1-8972-96c245d239e4 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 4.26 Windows x86-64 Interactive Installer cba59404-da4d-4009-8edb-6655b72464dd 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 3.25 Linux x86-32 Interactive Installer 0e139bf2-f0da-4ba9-bb40-9c0cadb4087e 0 70150000000rxXPAAY PostgreSQL
PostgreSQL Download 9 3.25 Linux x86-64 Interactive Installer 566673e2-7d5e-46aa-82dc-7400d296400e 0 70150000000rxXPAAY PostgreSQL
PostgreSQL Download 9 3.25 Mac OS X Interactive Installer 72c6d5f4-68bb-4080-a8cf-769f5fc60f41 0 70150000000rxXPAAY PostgreSQL
PostgreSQL Download 9 3.25 Windows x86-32 Interactive Installer 46c086f2-944c-49a2-bac3-204ccf14ac17 0 70150000000rxXPAAY PostgreSQL
PostgreSQL Download 9 3.25 Windows x86-64 Interactive Installer dbb5a7ac-4ecb-4a11-be2b-d56e8908b2a4 0 70150000000rxXPAAY PostgreSQL