Java 多线程 HTTP

Question

我有一个工作程序循环遍历 UID 的列表（20,000 多个项目），构建、连接、序列化和保存找到的项目属性。工作正常。

我想实现的是加快速度。它必须发出的那 20,000 多个 HTTP 请求以及之后的所有请求.. 它不是特别快。

我试过阅读多线程和下面的代码，关于 connectionManager。重新使用 HttpClient 等。但我无法理解或将给定的代码应用于我的情况。

如何创建我的代码，使其同时发送多个 HTTP 请求以加快处理速度？

PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
        CloseableHttpClient httpClient = HttpClients.custom()
                .setConnectionManager(cm)
                .build();

下面是我当前的代码，我怎样才能使这个过程更快？

JSONObject httpJSONObject;
            for (int i = 0; i < missingUIDList.size(); i++)
                try {
                    HttpGet get = new HttpGet("https://api.guildwars2.com/v2/items/" + missingUIDList.get(i));
                    HttpClient client = HttpClientBuilder.create().build();
                    HttpResponse response = client.execute(get);
                    HttpEntity entity = response.getEntity();
                    String result = EntityUtils.toString(entity);
                    httpJSONObject = new JSONObject(result);

                    itemRoot items = new Gson().fromJson(httpJSONObject.toString(), itemRoot.class);
                    String name = items.getName().replaceAll("'","''");
                    connection = DriverManager.getConnection("jdbc:sqlite:gw2.db");
                    Statement statement = connection.createStatement();
                    statement.setQueryTimeout(30);  // set timeout to 30 sec.
                    String cookie = "INSERT INTO item VALUES('" + name +
                            "','" + items.getDescription() +
                            "','" + items.getType() +
                            "'," + items.getLevel() +
                            ",'" + items.getRarity() +
                            "'," + items.getVendor_value() +
                            "," + items.getDefault_skin() +
                            "," + items.getId() +
                            ",'" + items.getChat_link() +
                            "','" + items.getIcon() +
                            "');";
                    System.out.println(cookie);
                    statement.executeUpdate(cookie);
                } catch (ClientProtocolException e) {
                    e.printStackTrace();
                } catch (IOException e) {
                    e.printStackTrace();
                } catch (JSONException e) {
                    e.printStackTrace();
                } catch (SQLException e) {
                    System.err.println(e.getMessage());
                }
        }

编辑：

根据 Vadim 的提示，这是针对单线程优化的代码，希望更多。

private void addMissingItems(List<Integer> missingUIDList) {
    HttpClient client = HttpClientBuilder.create().build();
    HttpResponse response;
    HttpEntity entity;
    String result;
    try {
        connection = DriverManager.getConnection("jdbc:sqlite:gw2.db");
        statement = connection.createStatement();
        statement.setQueryTimeout(30);  // set timeout to 30 sec.
    } catch (SQLException e) {
        System.err.println(e.getMessage());
    }

    for (int i = 0; i < missingUIDList.size(); i++)
        try {
            HttpGet get = new HttpGet("https://api.guildwars2.com/v2/items/" + missingUIDList.get(i));
            response = client.execute(get);
            entity = response.getEntity();
            result = EntityUtils.toString(entity);
            JSONObject httpJSONObject = new JSONObject(result);
            itemRoot items = new Gson().fromJson(httpJSONObject.toString(), itemRoot.class);

            System.out.println(httpJSONObject.getInt("id"));
            String cookie = "INSERT INTO item VALUES('" + items.getName().replaceAll("'","''") +
                    "','" + items.getDescription() +
                    "','" + items.getType() +
                    "'," + items.getLevel() +
                    ",'" + items.getRarity() +
                    "'," + items.getVendor_value() +
                    "," + items.getDefault_skin() +
                    "," + items.getId() +
                    ",'" + items.getChat_link() +
                    "','" + items.getIcon() +
                    "');";
            statement.executeUpdate(cookie);

        } catch (ClientProtocolException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        } catch (JSONException e) {
            e.printStackTrace();
        } catch (SQLException e) {
            System.err.println(e.getMessage());
        }
}

Answer 1

解决方案使用Executor services

PoolingHttpClientConnectionManager cm = new   PoolingHttpClientConnectionManager();
CloseableHttpClient httpClient = HttpClients.custom()
            .setConnectionManager(cm)
            .build();

private final ExecutorService pool = Executors.newFixedThreadPool(poolSize);

for (int i = 0; i < missingUIDList.size(); i++) {
    HttpGet get = new HttpGet("https://api.guildwars2.com/v2/items/" + missingUIDList.get(i));
    pool.execute(new Worker(get));
}

class Worker implements Runnable {
    private final HttpGet get;
    private final CloseableHttpClient httpClient;
    Handler(CloseableHttpClient httpClient,HttpGet get) { 
        this.get = get;
        this.httpClient = httpClient;
    }
    public void run() {
        try {
            HttpResponse response = client.execute(get);
            HttpEntity entity = response.getEntity();
            String result = EntityUtils.toString(entity);
            httpJSONObject = new JSONObject(result);
            ....
            //rest of your code
            ....
            statement.executeUpdate(cookie);
        } catch (ClientProtocolException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        } catch (JSONException e) {
            e.printStackTrace();
        } catch (SQLException e) {
            System.err.println(e.getMessage());
        }
    }
}

Answer 2

我可以建议先优化您现有的单线程代码，然后再进入多线程吗？之后转为多线程就容易多了。

您的 for 循环中有两个部分：

HTTP 调用数据
数据库调用存储数据

对于这两个部分，您需要通过打开新连接来执行非常耗时的操作。

相反，您可以：对于 http 部分（至少），像这样将客户端创建移出循环：

HttpClient client = HttpClientBuilder.create().build();
HttpResponse response;
HttpEntity entity;
String result;

然后在循环中重复使用它们：

for (int i = 0; i < missingUIDList.size(); i++)
 try {
       HttpGet get = new HttpGet("https://api.guildwars2.com/v2/items/" + missingUIDList.get(i));
       response = client.execute(get);
       entity = response.getEntity();
       result = EntityUtils.toString(entity);
       httpJSONObject = new JSONObject(result);
                ...

对于数据库部分（至少），

将连接创建移出循环（类似于上面
使用参数而不是连接值进行 INSERT SQL（永远不要那样做 - SQL 世界上存在注入）
也在循环外创建 PreparedStatement
在循环内设置参数并一遍又一遍地执行相同的查询。

可选地，有许多不同的方法来制作批量插入，它在一个数据库调用中插入许多记录，而不是运行然后一条一条地插入。

Java 多线程 HTTP

Java multithreading HTTP

java

connection

multithreading

http