如何在 C# 中实现分页逻辑？

Question

我正在从具有内置分页功能的 API 请求数据。我成功地能够请求数据（一次 100 行）并将其插入到我的数据库中。在超过 100 行可用的情况下，我没有得到完整的数据集。

基本上分页是通过发出请求来处理的，只要返回“光标”就可以进行后续请求。

我调用任务：

static async Task Main(string[] args){...

var employees = await ProcessEmployees(userAndPasswordToken, BaseUrl);

我的任务是这样的：

private static async Task<List<Employees>> ProcessEmployees(
    string userAndPasswordToken, string BaseUrl)
{
    //Construct urls
    string RequestPath = string.Format("/employees");
    string FullUrl = string.Format("{0}{1}", BaseUrl, RequestPath);

    var streamTask = client.GetStreamAsync(FullUrl);
    var employees = await JsonSerializer.DeserializeAsync<List<Employees>>(
        await streamTask);

    return employees;
}

如果返回的行数超过 100 行，如何检查 header 游标然后多次插入？

供应商提供的分页示例代码：

// Paging is handled by making a request and then making
// follow up requests as long as a "cursor" is returned.
string cursor = null;
do
{
    var query = HttpUtility.ParseQueryString("");
    query["locationId"] = "1";
    query["businessDate"] = "2019-04-30";
    if (cursor != null) query["cursor"] = cursor;

    var fullUrl = $"{url}/{endpoint}?{query}";
    _testOutputHelper.WriteLine(fullUrl);
    var json = client.DownloadString(fullUrl);
    results.AddRange(
        JsonConvert.DeserializeObject<List<Check>>(json));

    cursor = client.ResponseHeaders["cursor"];
} while (cursor != null);

}

Answer 1

您的问题：如果返回的行超过 100 行，我如何检查 header 中的游标然后多次插入？

代码答案 :

private static async Task<List<Employees>> ProcessEmployees(string 
userAndPasswordToken, string BaseUrl)
{
    //Construct urls
    string RequestPath = string.Format("/employees");
    string FullUrl = string.Format("{0}{1}", BaseUrl, RequestPath);
    
    // use GetAsync instead of GetStreamAsync unless it's mandatory for your codebase.
    var response = await client.GetAsync(FullUrl);

    // you can extract string from the response right away
    var content = await response.Content.ReadAsStringAsync();

    // and pass it in instead of Steam.
    var employees = await JsonSerializer.DeserializeAsync<List<Employees>>(content);

    // place the header key mapped with the cursor value.
    IEnumerable<string> values = response.Headers.GetValues("{your-cursor-key}");

    // check if `values` above has a value. If it returns more than one value,
    // you need to figure out how to get the right value.
    var cursor = values?.FirstOrDefault();
    while(cursor != null) 
    {
        // You haven't provided any information about what the cursor really is.
        // I just assumed that it's just a simple query string with `cursor` key name.
        RequestPath += $"?cursor={cursor}";
        FullUrl = string.Format("{0}{1}", BaseUrl, RequestPath);
        response = await client.GetAsync(FullUrl);
        content = await response.Content.ReadAsStringAsync();
        var nextEmployees = await JsonSerializer.DeserializeAsync<List<Employees>>(content);

        // add employees in the next page.
        employees.AddRange(nextEmployees);
        
        values = response.Headers.GetValues("{your-cursor-key}");
        cursor = values?.FirstOrDefault();
    }

    return employees;
}

我真的不能给你一个确切的答案，因为你的问题中遗漏了太多细节。但看起来您想获得一些实际的代码片段。上面的代码片段无法运行，需要修改。

这完全取决于服务器如何向您发送带有 Cursor 的响应。

请注意...

如果在响应 header 中总是有 Cursor 值，例如）“0”或“N/A”，光标将不会是 null 并且您的客户端程序将永远保留运行 while 语句。
如果在调用/employees时在请求header中使用Cursor，则应该构建HttpRequestMessage并使用SendAsync。

这里是 HttpRequestMessage 和 SendAsync 示例。

var requestMessage = new HttpRequestMessage
{
    Method = HttpMethod.Get,
    RequestUri = new Uri(FullUrl)
};
requestMessage.Headers.Add("{acceptable-cursor-header-key}", cursor);
response = client.SendAsync(requestMessage);

Answer 2

在@hina10531 的帮助下，我成功地重写了任务。我现在运行遇到的问题是请求太多 (429)，因为我对此 API 的访问级别每分钟只允许 60 次调用 - 另一个问题，另一个 post.

回答如下：

        private static async Task<List<Employees>> ProcessEmployees(string userAndPasswordToken, string BaseUrl)
    {
        //Construct urls
        string RequestPath = string.Format("general/employees");
        string FullUrl = string.Format("{0}{1}", BaseUrl, RequestPath);
        string CursorPath = string.Format("");
        
        // use GetAsync instead of GetStreamAsync unless it's mandatory for your codebase.
        var response = await client.GetAsync(FullUrl);

        // you can extract string from the response right away
        var content = await response.Content.ReadAsStringAsync();
        
        // and pass it in instead of Steam.
        //var employees = JsonSerializer.DeserializeAsync<List<Employees>>(content);
        var employees = Newtonsoft.Json.JsonConvert.DeserializeObject<List<Employees>>(content);
        
        // place the header key mapped with the cursor value.
        IEnumerable<string> values = response.Headers.GetValues("cursor");

        // check if `values` above has a value. If it returns more than one value,
        // you need to figure out how to get the right value.
       
        string cursor = null;
        cursor = values?.FirstOrDefault();
        do
        {
            // You haven't provided any information about what the cursor really is.
            // I just assumed that it's just a simple query string with `cursor` key name.
            CursorPath = $"?cursor={cursor}";
            if (cursor == null) CursorPath = string.Format("");
            FullUrl = string.Format("{0}{1}{2}", BaseUrl, RequestPath, CursorPath);
            response = await client.GetAsync(FullUrl);
            Console.WriteLine(response);
            content = await response.Content.ReadAsStringAsync();
            var nextEmployees = Newtonsoft.Json.JsonConvert.DeserializeObject<List<Employees>>(content);

            // add employees in the next page.
            employees.AddRange(nextEmployees);
            
            values = response.Headers.GetValues("cursor");
            cursor = values?.FirstOrDefault();

        } while(cursor != null);

        return employees;
    }

如何在 C# 中实现分页逻辑？

How can I implement paging logic in C#?

c#

api

paging

cursor