直到数据完成才返回响应

Question

我在使用 Apache HttpClient4 通过 POST 操作读取大型响应流时遇到问题。正在发送响应，但长度未知。当我用 curl 使用它时，它会立即开始使用，但是 HttpClient 它会等到收到整个响应后才开始使用。

对于 curl，activity 看起来像这样，输出几乎立即开始写入：

$ curl -v -X POST --data-binary @input.gz http://chemservices:8080/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf  -H 'Content-Type: application/x-squonk-dataset-molecule+json' -H 'Content-Encoding: gzip' -H 'Accept: chemical/x-mdl-sdfile' -o output.sdf -v
Note: Unnecessary use of -X or --request, POST is already inferred.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 172.18.0.6...
* TCP_NODELAY set
* Connected to chemservices (172.18.0.6) port 8080 (#0)
> POST /chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf HTTP/1.1
> Host: chemservices:8080
> User-Agent: curl/7.58.0
> Content-Type: application/x-squonk-dataset-molecule+json
> Content-Encoding: gzip
> Accept: chemical/x-mdl-sdfile
> Content-Length: 19397182
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
} [16384 bytes data]
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Server: Apache-Coyote/1.1
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET, HEAD, POST, PUT, DELETE, TRACE, OPTIONS, CONNECT, PATCH
< Access-Control-Max-Age: 3600
< Access-Control-Allow-Headers: Origin, Accept, X-Requested-With, Content-Type, Access-Control-Request-Method, Access-Control-Request-Headers
< Content-Type: chemical/x-mdl-sdfile
< Transfer-Encoding: chunked
< Date: Sun, 13 Jan 2019 12:30:18 GMT
< 
{ [1031 bytes data]
100  172M    0  154M  100 18.4M  7103k   851k  0:00:22  0:00:22 --:--:-- 7301k
* Connection #0 to host chemservices left intact

但是当我使用 HttpClient 时，execute() 操作会阻塞，直到整个响应被写入：

LOG.info("Posting commencing");
CloseableHttpResponse resp = httpclient.execute(httpPost);
LOG.info("Posting complete");

正在根据请求设置 Content-Type、Content-Encoding、Accept 和 Accept-Encoding headers。

如果我将 SocketTimeout 参数设置为足够大的值，我可以获得响应，但显然这不是正确的解决方案！

关于如何正确处理这个问题有什么建议吗？

Answer 1

我相信您将需要使用HttpAsyncClient来实现异步请求处理。

考虑以下示例，它将以块的形式流式传输响应，您可以在覆盖的 onCharReceived 方法中处理这些块。为了这个例子的目的，我只是让它打印每个块的长度。

显然您需要修改端点，headers 并根据需要请求数据：

import org.apache.http.HttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.concurrent.FutureCallback;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.nio.client.CloseableHttpAsyncClient;
import org.apache.http.impl.nio.client.HttpAsyncClients;
import org.apache.http.nio.IOControl;
import org.apache.http.nio.client.methods.AsyncCharConsumer;
import org.apache.http.nio.client.methods.HttpAsyncMethods;
import org.apache.http.nio.protocol.HttpAsyncRequestProducer;
import org.apache.http.protocol.HttpContext;

import java.nio.CharBuffer;
import java.util.concurrent.CountDownLatch;

public class HttpAsyncTest {

    public static void main(String[] args) {
        try (CloseableHttpAsyncClient httpclient = HttpAsyncClients.createDefault()) {
            httpclient.start();

            final CountDownLatch latch = new CountDownLatch(1);
            final HttpPost request = new HttpPost("https://postman-echo.com/post");
            request.setEntity(new StringEntity("This is the request data"));
            request.setHeader("Content-Type", "application/x-www-form-urlencoded");

            HttpAsyncRequestProducer producer = HttpAsyncMethods.create(request);
            AsyncCharConsumer<HttpResponse> consumer = new AsyncCharConsumer<HttpResponse>() {
                HttpResponse response;

                @Override
                protected void onResponseReceived(final HttpResponse response) {
                    this.response = response;
                }

                @Override
                protected void onCharReceived(final CharBuffer buf, final IOControl ioctrl) {
                    System.out.printf("onCharReceived: %d\n", buf.length());
                }

                @Override
                protected HttpResponse buildResult(final HttpContext context) {
                    return this.response;
                }
            };

            httpclient.execute(producer, consumer, new FutureCallback<HttpResponse>() {
                public void completed(final HttpResponse response3) {
                    latch.countDown();
                    System.out.println(request.getRequestLine() + "->" + response3.getStatusLine());
                }

                public void failed(final Exception ex) {
                    latch.countDown();
                    System.out.println(request.getRequestLine() + "->" + ex);
                }

                public void cancelled() {
                    latch.countDown();
                    System.out.println(request.getRequestLine() + " cancelled");
                }
            });

            latch.await();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

直到数据完成才返回响应

Response not returned until data is complete

java

apache-httpclient-4.x