异步 NancyFX 与 HttpClient - GetAsync for fbcdn returns 403 禁止？

Question

注意：我对这个问题做了很多改动，试图让它更能说明问题。下面的评论不再反映这个问题。

我正在尝试从 fbcdn 获取这张图片：

https://scontent.xx.fbcdn.net/v/t1.0-1/c15.0.50.50/p50x50/10354686_10150004552801856_220367501106153455_n.jpg?oh=6c801f82cd5a32fd6e5a4258ce00a314&oe=589AAD2F

浏览器可以正常使用。这是我的代码：

public class ReverseProxyController : NancyModule
{
    public ReverseProxyController()
    {
        Get["/", true] = async (parameters, ct) =>
        {
            var result = await GetResult(parameters, ct);
            return result;
        };
    }

    private async Task<Response> GetResult(dynamic parameters, CancellationToken ct)
    {
        var client = new HttpClient();
        string url = Request.Query["url"].Value.ToString();
        if (url == null) return null;

        client.DefaultRequestHeaders.Add("Access-Control-Allow-Origin", "*");
        client.DefaultRequestHeaders.Add("User-Agent",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36");
        client.DefaultRequestHeaders.Add("Upgrade-Insecure-Requests", "1");
        client.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
        client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, sdch, br");
        client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.8,ru;q=0.6");
        var response = await client.GetAsync(url, ct);

        ct.ThrowIfCancellationRequested();


        switch (response.StatusCode)
        {
            case HttpStatusCode.OK:
                var stream = await response.Content.ReadAsStreamAsync();

                return Response.FromStream(stream, response.Content.Headers.ContentType != null
                    ? response.Content.Headers.ContentType.ToString()
                    : "application/octet-stream");

            default:
                return Response.AsText("\nError " + response.StatusCode);
        }
    }
}

我每次都收到 403 Forbidden 响应。我认为添加 headers 可以使它起作用，但不行。

此代码适用于其他主机上的其他图像，例如：

https://s-media-cache-ak0.pinimg.com/564x/ff/f0/c9/fff0c988a4516659d4009f60e0694cb6.jpg

Answer 1

问题出在 URL 从 Nancy 请求中检索，没有通过 HttpClient 获取数据。

我假设您正在向 nancy 发送请求，例如：

http://localhost/?url=...

因此对于 Facebook，它将是：

http://localhost/?url=https://scontent.xx.fbcdn.net/v/t1.0-1/c15.0.50.50/p50x50/10354686_10150004552801856_220367501106153455_n.jpg?oh=6c801f82cd5a32fd6e5a4258ce00a314&oe=589AAD2F

但是对于这个 url string url = Request.Query["url"].Value.ToString(); 是不完整的并且缺少最后一部分 (&oe=589AAD2F)，因此服务器响应禁止。

下面是演示问题的简单更改：

private async Task<Response> GetResult(dynamic parameters, CancellationToken ct)
{
    var client = new HttpClient();
    var req = Request.Url.ToString();
    var queryStart = req.IndexOf("url=");
    if (queryStart == -1)
        return Nancy.HttpStatusCode.BadRequest;

    var url = req.Substring(queryStart + 4);
    if (string.IsNullOrEmpty(url))
        return Nancy.HttpStatusCode.BadRequest;

    client.DefaultRequestHeaders.Add("Access-Control-Allow-Origin", "*");
    client.DefaultRequestHeaders.Add("User-Agent",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36");
    client.DefaultRequestHeaders.Add("Upgrade-Insecure-Requests", "1");
    client.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
    client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, sdch, br");
    client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.8,ru;q=0.6");
    var response = await client.GetAsync(url, ct);

    ct.ThrowIfCancellationRequested();


    switch (response.StatusCode)
    {
        case System.Net.HttpStatusCode.OK:
            var stream = await response.Content.ReadAsStreamAsync();

            return Response.FromStream(stream, response.Content.Headers.ContentType != null
                ? response.Content.Headers.ContentType.ToString()
                : "application/octet-stream");

        default:
            return Response.AsText("\nError " + response.StatusCode);
    }
}

解决方案

我们实际上可以在发送之前对 URL 进行编码，Nancy 会自动为我们解码 URL，因此无需在服务器端进行任何更改。

这是一个 link 使用 HttpUtility.UrlEncode 生成的示例，应用于

https://scontent.xx.fbcdn.net/v/t1.0-1/c15.0.50.50/p50x50/10354686_10150004552801856_220367501106153455_n.jpg?oh=6c801f82cd5a32fd6e5a4258ce00a314&oe=589AAD2F

结果：

https%3a%2f%2fscontent.xx.fbcdn.net%2fv%2ft1.0-1%2fc15.0.50.50%2fp50x50%2f10354686_10150004552801856_220367501106153455_n.jpg%3foh%3d6c801f82cd5a32fd6e5a4258ce00a314%26oe% 3d589AAD2F

这个特定 link 的实际请求是：

http://localhost:9876/?url=https%3a%2f%2fscontent.xx.fbcdn.net%2fv%2ft1.0-1%2fc15.0.50.50%2fp50x50%2f10354686_10150004552801856_220367501106153455_n.jpg%3foh%3d6c801f82cd5a32fd6e5a4258ce00a314%26oe%3d589AAD2F

备用解决方案

在这种情况下，我个人更喜欢 POST 而不是 GET，所以这里是：

public class ReverseProxyController : NancyModule
{
    class ProxyRequest
    {
        public string Url { get; set; }
    }

    public ReverseProxyController()
    {
        Post["/", true] = async (parameters, ct) =>
        {
            var result = await GetResult(parameters, ct);
            return result;
        };
    }

    private async Task<Response> GetResult(dynamic parameters, CancellationToken ct)
    {
        var pReq = this.Bind<ProxyRequest>();
        var url = pReq.Url;
        if (string.IsNullOrEmpty(url))
            return null;

        var client = new HttpClient();
        client.DefaultRequestHeaders.Add("Access-Control-Allow-Origin", "*");
        client.DefaultRequestHeaders.Add("User-Agent",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36");
        client.DefaultRequestHeaders.Add("Upgrade-Insecure-Requests", "1");
        client.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
        client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, sdch, br");
        client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.8,ru;q=0.6");
        var response = await client.GetAsync(url, ct);

        ct.ThrowIfCancellationRequested();


        switch (response.StatusCode)
        {
            case System.Net.HttpStatusCode.OK:
                var stream = await response.Content.ReadAsStreamAsync();

                return Response.FromStream(stream, response.Content.Headers.ContentType != null
                    ? response.Content.Headers.ContentType.ToString()
                    : "application/octet-stream");

            default:
                return Response.AsText("\nError " + response.StatusCode);
        }
    }
}

Answer 2

作为控制台应用程序，您编写的代码工作正常

   static void Main(string[] args)
    {
        GetStuff();
        Console.ReadLine();
    }

    private static async void GetStuff()
    {
        CancellationToken ct = new CancellationToken();
        var client = new HttpClient();
        //this is only a Simple Change to demonstrate the problem, and should not be considered as a proper solution

        string url = "https://scontent.xx.fbcdn.net/v/t1.0-1/c15.0.50.50/p50x50/10354686_10150004552801856_220367501106153455_n.jpg?oh=6c801f82cd5a32fd6e5a4258ce00a314&oe=589AAD2F";
        if (url == null) return;

        client.DefaultRequestHeaders.Add("Access-Control-Allow-Origin", "*");
        client.DefaultRequestHeaders.Add("User-Agent",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36");
        client.DefaultRequestHeaders.Add("Upgrade-Insecure-Requests", "1");
        client.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
        client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, sdch, br");
        client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.8,ru;q=0.6");
        var response = await client.GetAsync(url, ct);

        ct.ThrowIfCancellationRequested();


        switch (response.StatusCode)
        {
            case System.Net.HttpStatusCode.OK:
                var stream = await response.Content.ReadAsStreamAsync();


                break;
            default:
                break;
        }
    }

我收到一条 OK 200 return 消息。您使用的其他图片是否需要登录信息？

异步 NancyFX 与 HttpClient - GetAsync for fbcdn returns 403 禁止？

Async NancyFX with HttpClient - GetAsync for fbcdn returns 403 forbidden?

.net

c#

async-await

nancy

dotnet-httpclient