反正有没有使用 "BrowserSession" 下载文件？ C＃

Question

我有一个网站需要先登录才能让您下载文件。目前我正在使用 BrowserSession Class 登录并执行所有所需的抓取操作（至少在大多数情况下）。

BrowserSession Class 来源在 post 底部：

下载链接显示在文档节点上。但是我不知道如何向 class 添加下载功能，如果我尝试使用网络客户端下载它们会失败，我已经不得不大量修改 BrowserSession class，（我应该将其修改为部分但没有）所以我真的不想改变使用 BrowserSession Class.

我相信它使用 htmlAgilityPack.HtmlWeb 下载和加载网页。

如果没有简单的方法来修改 BrowserSession，是否有办法将它的 CookieCollection 与 Webclient 一起使用？

PS: 我需要登录才能下载文件，否则 link 会重定向到登录屏幕。这就是为什么我无法简单地使用 WebClient，并且需要修改 BrowserSession class 才能下载，或者修改 WebClient 以在获取页面之前使用 cookie。

我承认我不太了解 cookies（我不确定是每次使用 GET 时都使用它们，还是只是在 POST 上），但到目前为止 BrowserSession 已经处理好了所有这些。

PPS:我发的这个BrowserSession也不是我加的，但是核心功能都是一样的

public class BrowserSession
{
private bool _isPost;
private HtmlDocument _htmlDoc;

/// <summary>
/// System.Net.CookieCollection. Provides a collection container for instances of Cookie class 
/// </summary>
public CookieCollection Cookies { get; set; }

/// <summary>
/// Provide a key-value-pair collection of form elements 
/// </summary>
public FormElementCollection FormElements { get; set; }

/// <summary>
/// Makes a HTTP GET request to the given URL
/// </summary>
public string Get(string url)
{
    _isPost = false;
    CreateWebRequestObject().Load(url);
    return _htmlDoc.DocumentNode.InnerHtml;
}

/// <summary>
/// Makes a HTTP POST request to the given URL
/// </summary>
public string Post(string url)
{
    _isPost = true;
    CreateWebRequestObject().Load(url, "POST");
    return _htmlDoc.DocumentNode.InnerHtml;
}

/// <summary>
/// Creates the HtmlWeb object and initializes all event handlers. 
/// </summary>
private HtmlWeb CreateWebRequestObject()
{
    HtmlWeb web = new HtmlWeb();
    web.UseCookies = true;
    web.PreRequest = new HtmlWeb.PreRequestHandler(OnPreRequest);
    web.PostResponse = new HtmlWeb.PostResponseHandler(OnAfterResponse);
    web.PreHandleDocument = new HtmlWeb.PreHandleDocumentHandler(OnPreHandleDocument);
    return web;
}

/// <summary>
/// Event handler for HtmlWeb.PreRequestHandler. Occurs before an HTTP request is executed.
/// </summary>
protected bool OnPreRequest(HttpWebRequest request)
{
    AddCookiesTo(request);               // Add cookies that were saved from previous requests
    if (_isPost) AddPostDataTo(request); // We only need to add post data on a POST request
    return true;
}

/// <summary>
/// Event handler for HtmlWeb.PostResponseHandler. Occurs after a HTTP response is received
/// </summary>
protected void OnAfterResponse(HttpWebRequest request, HttpWebResponse response)
{
    SaveCookiesFrom(response); // Save cookies for subsequent requests
}

/// <summary>
/// Event handler for HtmlWeb.PreHandleDocumentHandler. Occurs before a HTML document is handled
/// </summary>
protected void OnPreHandleDocument(HtmlDocument document)
{
    SaveHtmlDocument(document);
}

/// <summary>
/// Assembles the Post data and attaches to the request object
/// </summary>
private void AddPostDataTo(HttpWebRequest request)
{
    string payload = FormElements.AssemblePostPayload();
    byte[] buff = Encoding.UTF8.GetBytes(payload.ToCharArray());
    request.ContentLength = buff.Length;
    request.ContentType = "application/x-www-form-urlencoded";
    System.IO.Stream reqStream = request.GetRequestStream();
    reqStream.Write(buff, 0, buff.Length);
}

/// <summary>
/// Add cookies to the request object
/// </summary>
private void AddCookiesTo(HttpWebRequest request)
{
    if (Cookies != null && Cookies.Count > 0)
    {
        request.CookieContainer.Add(Cookies);
    }
}

/// <summary>
/// Saves cookies from the response object to the local CookieCollection object
/// </summary>
private void SaveCookiesFrom(HttpWebResponse response)
{
    if (response.Cookies.Count > 0)
    {
        if (Cookies == null)  Cookies = new CookieCollection(); 
        Cookies.Add(response.Cookies);
    }
}

/// <summary>
/// Saves the form elements collection by parsing the HTML document
/// </summary>
private void SaveHtmlDocument(HtmlDocument document)
{
    _htmlDoc = document;
    FormElements = new FormElementCollection(_htmlDoc);
}
}

FormElementCollection Class:

/// <summary>
/// Represents a combined list and collection of Form Elements.
/// </summary>
public class FormElementCollection : Dictionary<string, string>
{
/// <summary>
/// Constructor. Parses the HtmlDocument to get all form input elements. 
/// </summary>
public FormElementCollection(HtmlDocument htmlDoc)
{
    var inputs = htmlDoc.DocumentNode.Descendants("input");
    foreach (var element in inputs)
    {
        string name = element.GetAttributeValue("name", "undefined");
        string value = element.GetAttributeValue("value", "");
        if (!name.Equals("undefined")) Add(name, value);
    }
}

/// <summary>
/// Assembles all form elements and values to POST. Also html encodes the values.  
/// </summary>
public string AssemblePostPayload()
{
    StringBuilder sb = new StringBuilder();
    foreach (var element in this)
    {
        string value = System.Web.HttpUtility.UrlEncode(element.Value);
        sb.Append("&" + element.Key + "=" + value);
    }
    return sb.ToString().Substring(1);
}
}

Answer 1

登录和下载网页并不容易。我最近遇到了同样的问题。如果您从中找到解决方案，请提供。

现在我所做的是将 Selenium 与 PhantomJS 一起使用。使用 Selenium，我可以与我选择的网络浏览器进行交互。

此外，浏览器 class 不使用 Html Agility Pack，它是可通过 nuget 获得的第三方库。

我想向您推荐这个，我在其中创建了一个完整示例，说明如何使用 Selenium 以及如何下载 Html 文档并使用 xpath 过滤掉必要的信息。

Answer 2

我设法让它工作，使用 BrowserSession 和修改后的 webClient:

首先将 _htmlDoc 更改为 Public 以访问文档节点：

public class BrowserSession
{
    private bool _isPost;
    public string previous_Response { get; private set; }
    public HtmlDocument _htmlDoc { get; private set; }
}

其次将此方法添加到 BrowserSession:

 public void DownloadCookieProtectedFile(string url, string Filename)
    {
        using (CookieAwareWebClient wc = new CookieAwareWebClient())
        {
            wc.Cookies = Cookies;
            wc.DownloadFile(url, Filename);
        }
    }
//rest of BrowserSession

第三次添加此 Class 某处，允许将 cookie 从 BrowserSession 传递到 WebClient。

public class CookieAwareWebClient : WebClient
{
    public CookieCollection Cookies = new CookieCollection();
    private void AddCookiesTo(HttpWebRequest request)
    {
        if (Cookies != null && Cookies.Count > 0)
        {
            request.CookieContainer.Add(Cookies);
        }
    }

    protected override WebRequest GetWebRequest(Uri address)
    {
        WebRequest request = base.GetWebRequest(address);
        HttpWebRequest webRequest = request as HttpWebRequest;
        if (webRequest != null)
        {
            if (webRequest.CookieContainer == null) webRequest.CookieContainer = new CookieContainer();
            AddCookiesTo(webRequest);
        }
        return request;
    }
}

这应该使您能够像往常一样使用 BrowserSession，并且当您需要获取一个您只能访问的文件时如果您已登录，只需调用 BrowserSession.DownloadCookieProtectedFile() 就好像它是一个 WebClient，只像这样设置 Cookies:

Using(wc = new CookieAwareWebClient())
{
    wc.Cookies = BrowserSession.Cookies
    //Download with WebClient As normal
    wc.DownloadFile();
}

反正有没有使用 "BrowserSession" 下载文件？ C＃

Is there anyway to use "BrowserSession" to download files? C#

c#

cookies

download

html-agility-pack