ScrapySharp 表单提交导致 System.AggregateException

Question

我花了几个小时绞尽脑汁想知道为什么这不起作用

我正在尝试使用 ScrapySharp 来抓取网站，现在只是试用示例网站，然后转到我的实际网站。

每次我在程序中执行 form.Submit() 时，我都会遇到 System.AggregateException（指定的转换无效）

我的代码：

using System;
using System.IO;
using System.Linq;
using System.Net;
using HtmlAgilityPack;
using ScrapySharp.Extensions;
using ScrapySharp.Html;
using ScrapySharp.Html.Forms;
using ScrapySharp.Network;

namespace WebScraper
{
    class MainClass
    {
        public static void Main(string[] args)
        {
            ScrapingBrowser browser = new ScrapingBrowser();

            //set UseDefaultCookiesParser as false if a website returns invalid cookies format
            //browser.UseDefaultCookiesParser = false;
            browser.AllowAutoRedirect = true;
            browser.AllowMetaRedirect = true;
            WebPage homePage = browser.NavigateToPage(new Uri("http://the-internet.herokuapp.com/login"));

            PageWebForm form = homePage.FindForm("login");
            form["username"] = "tomsmith";
            form["password"] = "SuperSecretPassword!";
            form.Method = HttpVerb.Get; //I tried both .Post and .Get
            WebPage resultsPage = form.Submit(); //THIS IS WHERE I GET THE ERROR
            Console.WriteLine(resultsPage);

        }
    }
}

我的错误：

System.AggregateException: One or more errors occurred. (Specified cast is not valid.) ---> System.InvalidCastException: Specified cast is not valid. at ScrapySharp.Network.ScrapingBrowser.CreateRequest (System.Uri url, ScrapySharp.Network.HttpVerb verb) [0x0000b] in <0a639adc663f45108f057c429262c620>:0 at ScrapySharp.Network.ScrapingBrowser.NavigateToPageAsync (System.Uri url, ScrapySharp.Network.HttpVerb verb, System.String data, System.String contentType) [0x00066] in <0a639adc663f45108f057c429262c620>:0 --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional (System.Boolean includeTaskCanceledExceptions) [0x00011] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-06/external/bockbuild/builds/mono-x64/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:2027 at System.Threading.Tasks.Task1[TResult].GetResultCore (System.Boolean waitCompletionNotification) [0x0002b] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-06/external/bockbuild/builds/mono-x64/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:496 at System.Threading.Tasks.Task1[TResult].get_Result () [0x00000] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-06/external/bockbuild/builds/mono-x64/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:466 at ScrapySharp.Network.ScrapingBrowser.NavigateToPage (System.Uri url, ScrapySharp.Network.HttpVerb verb, System.String data, System.String contentType) [0x0000b] in <0a639adc663f45108f057c429262c620>:0 at ScrapySharp.Html.Forms.PageWebForm.Submit () [0x00023] in <0a639adc663f45108f057c429262c620>:0 at WebScraper.MainClass.Main (System.String[] args) [0x00065] in /Users/arib/Projects/WebScraper/WebScraper/Program.cs:29

我已经厌倦了这个错误，非常感谢任何和所有的帮助.. 谢谢

Answer 1

问题是当您使用 form["username"] 时，结果是一个字符串。您想要获得 FormField，您可以使用以下代码：

WebPage homePage = browser.NavigateToPage(new Uri("http://the-internet.herokuapp.com/login"));
PageWebForm form = homePage.FindForm("login");
var formFields = form.FormFields;
foreach (var field in formFields)
{
    if (field.Name.Equals("username", StringComparison.OrdinalIgnoreCase))
    {
        field.Value = "tomsmith";

    }
    else if (field.Name.Equals("password", StringComparison.OrdinalIgnoreCase))
    {
        field.Value = "SuperSecretPassword!";

    }
}

WebPage resultsPage = form.Submit();
Console.WriteLine(resultsPage);

或者，您可以使用 Find() 来获取 FormField:

var usernameField = form.FormFields.Find(x => x.Name == "username");

ScrapySharp 表单提交导致 System.AggregateException

ScrapySharp Form Submit causing System.AggregateException

.net

c#

web-scraping

aggregateexception

scrapysharp