AWS S3 v2 GetObject 无法解组响应 (java.net.SocketTimeoutException)
AWS S3 v2 GetObject Unable to unmarshall response (java.net.SocketTimeoutException)
我正在尝试从 S3 获取一个大对象并将输出直接传输到 HttpServletResponse 输出流。
然而，由于 SocketTimeoutException.
，它在这个对象上总是失败
这是使用 S3Client 获取对象的代码：
public void downloadMergedResults(final String jobId, final OutputStream os) {
get("jobs/" + jobId + "/results", os);
}
private boolean get(final String key, final OutputStream outputStream) {
log.trace("GET {}", key);
try {
client.getObject(GetObjectRequest.builder().bucket(bucket).key(key).build(), ResponseTransformer.toOutputStream(outputStream));
return true;
} catch (S3Exception e) {
if (e.statusCode() == 404) {
log.info("Item with key: {} does not exist.", key);
}
else {
log.error("Could not get item {} from S3. Error: {} {}", key, e.getMessage(), e.awsErrorDetails());
}
}
return false;
}
以下是我在 spring 启动应用程序中使用它的方式：
public Callable<ResponseEntity<StreamingResponseBody>> allData(HttpServletResponse res, @PathVariable("jobId") String jobId) {
Progress progress = s3client.downloadCachedProgress(jobId);
if (progress == null) {
log.info("{} Could not find progress file", jobId);
return () -> new ResponseEntity<>(HttpStatus.NOT_FOUND);
}
res.setContentType(MediaType.APPLICATION_JSON_VALUE);
res.setContentLengthLong(OPEN_ARRAY.length + progress.getDataSizeBytes() + (SEPARATOR.length * (progress.getMediaCount() - 1)) + CLOSE_ARRAY.length);
return () -> {
StreamingResponseBody streamingResponseBody = out -> {
try {
s3client.downloadMergedResults(jobId, out);
} catch(Exception e) {
log.error("{} Error downloading results: {}", jobId, e.getMessage(), e);
}
};
return new ResponseEntity<>(streamingResponseBody, HttpStatus.OK);
};
}
curl 是这么说的：
curl: (18) transfer closed with 524076376 bytes remaining to read
这是我的应用程序的堆栈跟踪：
software.amazon.awssdk.core.exception.SdkClientException: Unable to unmarshall response (java.net.SocketTimeoutException). Response Code: 200, Response Text: OK
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleSuccessResponse(CombinedResponseHandler.java:108) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:72) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:189) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:121) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:147) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:62) ~[aws-core-2.13.44.jar!/:na]
at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:3539) ~[s3-2.13.44.jar!/:na]
at org.me.blobstorage.S3ClientWrapper.get(S3ClientWrapper.java:362) ~[classes!/:na]
at org.me.blobstorage.S3ClientWrapper.downloadMergedResults(S3ClientWrapper.java:253) ~[classes!/:na]
at org.me.RequestHandler.lambda$null(RequestHandler.java:241) ~[classes!/:na]
at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:111) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:98) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.context.request.async.WebAsyncManager.lambda$startCallableProcessing(WebAsyncManager.java:328) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_265]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_265]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_265]
Caused by: software.amazon.awssdk.core.exception.NonRetryableException: java.net.SocketTimeoutException
at software.amazon.awssdk.core.exception.NonRetryableException$BuilderImpl.build(NonRetryableException.java:92) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.transformResponse(BaseSyncClientHandler.java:187) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.handle(BaseSyncClientHandler.java:168) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleSuccessResponse(CombinedResponseHandler.java:97) ~[sdk-core-2.13.44.jar!/:na]
... 40 common frames omitted
Caused by: org.apache.catalina.connector.ClientAbortException: java.net.SocketTimeoutException
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:776) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:681) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:386) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:364) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at software.amazon.awssdk.utils.IoUtils.copy(IoUtils.java:113) ~[utils-2.13.44.jar!/:na]
at software.amazon.awssdk.utils.IoUtils.copy(IoUtils.java:99) ~[utils-2.13.44.jar!/:na]
at software.amazon.awssdk.core.sync.ResponseTransformer.lambda$toOutputStream(ResponseTransformer.java:159) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.transformResponse(BaseSyncClientHandler.java:180) ~[sdk-core-2.13.44.jar!/:na]
... 42 common frames omitted
Caused by: java.net.SocketTimeoutException: null
at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:134) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:152) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1257) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:814) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:584) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:522) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:538) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:73) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:190) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.Response.doWrite(Response.java:601) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:339) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
... 51 common frames omitted
以下是我尝试初始化 S3Client 的不同方法：
1:
client = S3Client.builder()
.region(region)
.credentialsProvider(credentialsProvider)
.build();
2:
client = S3Client.builder()
.region(region)
.overrideConfiguration(ClientOverrideConfiguration.builder()
.apiCallTimeout(Duration.ofHours(1))
.build())
.credentialsProvider(credentialsProvider)
.build();
3:
client = S3Client.builder()
.region(region)
.overrideConfiguration(ClientOverrideConfiguration.builder()
.apiCallTimeout(Duration.ofHours(1))
.build())
.httpClient(UrlConnectionHttpClient.builder()
.connectionTimeout(Duration.ofHours(1))
.socketTimeout(Duration.ofHours(1)).build())
.credentialsProvider(credentialsProvider)
.build();
4:
client = S3Client.builder()
.region(region)
.overrideConfiguration(ClientOverrideConfiguration.builder()
.apiCallTimeout(Duration.ofHours(1))
.build())
.httpClientBuilder(UrlConnectionHttpClient.builder()
.connectionTimeout(Duration.ofHours(1))
.socketTimeout(Duration.ofHours(1)))
.credentialsProvider(credentialsProvider)
.build();
我的 spring 启动 applications.properties 请求超时设置：
spring.mvc.async.request-timeout=-1
我的 bootstrap class 上有 @EnableAsync。
请求通过 nginx 代理，位置设置为：
upstream myupstream {
server [SERVER-REDACTED]:[PORT-REDACTED];
keepalive 15;
}
server {
...
location ~ ^/(v1/bigsearch|actuator)/ {
proxy_pass http://myupstream;
proxy_http_version 1.1;
proxy_set_header Connection "Keep-Alive";
proxy_set_header Proxy-Connection "Keep-Alive";
client_max_body_size 2G;
}
...
}
以及相关的 nginx.conf 条目本身：
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 1d;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
proxy_read_timeout 1d;
proxy_send_timeout 1d;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
gzip on;
gzip_disable "msie6";
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
问题
我还没有做些什么来完成这项工作？
一如既往，我为一个问题苦苦挣扎了好几天，post 它在 SO 上，然后弄清楚。
即使错误表现为 SocketTimeoutException，问题实际上出在 nginx 代理缓冲上。
一段时间后，我觉得下载量总是超过 1G 很重要，直到我专门搜索它，我才在这里发现实际问题和解决方案：https://github.com/owncloud/client/issues/5706
将这些设置添加到 nginx.conf 文件中解决了它：
proxy_request_buffering off;
proxy_buffering off;
我正在尝试从 S3 获取一个大对象并将输出直接传输到 HttpServletResponse 输出流。
然而，由于 SocketTimeoutException.
这是使用 S3Client 获取对象的代码：
public void downloadMergedResults(final String jobId, final OutputStream os) {
get("jobs/" + jobId + "/results", os);
}
private boolean get(final String key, final OutputStream outputStream) {
log.trace("GET {}", key);
try {
client.getObject(GetObjectRequest.builder().bucket(bucket).key(key).build(), ResponseTransformer.toOutputStream(outputStream));
return true;
} catch (S3Exception e) {
if (e.statusCode() == 404) {
log.info("Item with key: {} does not exist.", key);
}
else {
log.error("Could not get item {} from S3. Error: {} {}", key, e.getMessage(), e.awsErrorDetails());
}
}
return false;
}
以下是我在 spring 启动应用程序中使用它的方式：
public Callable<ResponseEntity<StreamingResponseBody>> allData(HttpServletResponse res, @PathVariable("jobId") String jobId) {
Progress progress = s3client.downloadCachedProgress(jobId);
if (progress == null) {
log.info("{} Could not find progress file", jobId);
return () -> new ResponseEntity<>(HttpStatus.NOT_FOUND);
}
res.setContentType(MediaType.APPLICATION_JSON_VALUE);
res.setContentLengthLong(OPEN_ARRAY.length + progress.getDataSizeBytes() + (SEPARATOR.length * (progress.getMediaCount() - 1)) + CLOSE_ARRAY.length);
return () -> {
StreamingResponseBody streamingResponseBody = out -> {
try {
s3client.downloadMergedResults(jobId, out);
} catch(Exception e) {
log.error("{} Error downloading results: {}", jobId, e.getMessage(), e);
}
};
return new ResponseEntity<>(streamingResponseBody, HttpStatus.OK);
};
}
curl 是这么说的：
curl: (18) transfer closed with 524076376 bytes remaining to read
这是我的应用程序的堆栈跟踪：
software.amazon.awssdk.core.exception.SdkClientException: Unable to unmarshall response (java.net.SocketTimeoutException). Response Code: 200, Response Text: OK
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleSuccessResponse(CombinedResponseHandler.java:108) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:72) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:189) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:121) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:147) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:62) ~[aws-core-2.13.44.jar!/:na]
at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:3539) ~[s3-2.13.44.jar!/:na]
at org.me.blobstorage.S3ClientWrapper.get(S3ClientWrapper.java:362) ~[classes!/:na]
at org.me.blobstorage.S3ClientWrapper.downloadMergedResults(S3ClientWrapper.java:253) ~[classes!/:na]
at org.me.RequestHandler.lambda$null(RequestHandler.java:241) ~[classes!/:na]
at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:111) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:98) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.context.request.async.WebAsyncManager.lambda$startCallableProcessing(WebAsyncManager.java:328) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_265]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_265]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_265]
Caused by: software.amazon.awssdk.core.exception.NonRetryableException: java.net.SocketTimeoutException
at software.amazon.awssdk.core.exception.NonRetryableException$BuilderImpl.build(NonRetryableException.java:92) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.transformResponse(BaseSyncClientHandler.java:187) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.handle(BaseSyncClientHandler.java:168) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleSuccessResponse(CombinedResponseHandler.java:97) ~[sdk-core-2.13.44.jar!/:na]
... 40 common frames omitted
Caused by: org.apache.catalina.connector.ClientAbortException: java.net.SocketTimeoutException
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:776) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:681) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:386) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:364) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at software.amazon.awssdk.utils.IoUtils.copy(IoUtils.java:113) ~[utils-2.13.44.jar!/:na]
at software.amazon.awssdk.utils.IoUtils.copy(IoUtils.java:99) ~[utils-2.13.44.jar!/:na]
at software.amazon.awssdk.core.sync.ResponseTransformer.lambda$toOutputStream(ResponseTransformer.java:159) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.transformResponse(BaseSyncClientHandler.java:180) ~[sdk-core-2.13.44.jar!/:na]
... 42 common frames omitted
Caused by: java.net.SocketTimeoutException: null
at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:134) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:152) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1257) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:814) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:584) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:522) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:538) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:73) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:190) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.Response.doWrite(Response.java:601) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:339) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
... 51 common frames omitted
以下是我尝试初始化 S3Client 的不同方法：
1:
client = S3Client.builder()
.region(region)
.credentialsProvider(credentialsProvider)
.build();
2:
client = S3Client.builder()
.region(region)
.overrideConfiguration(ClientOverrideConfiguration.builder()
.apiCallTimeout(Duration.ofHours(1))
.build())
.credentialsProvider(credentialsProvider)
.build();
3:
client = S3Client.builder()
.region(region)
.overrideConfiguration(ClientOverrideConfiguration.builder()
.apiCallTimeout(Duration.ofHours(1))
.build())
.httpClient(UrlConnectionHttpClient.builder()
.connectionTimeout(Duration.ofHours(1))
.socketTimeout(Duration.ofHours(1)).build())
.credentialsProvider(credentialsProvider)
.build();
4:
client = S3Client.builder()
.region(region)
.overrideConfiguration(ClientOverrideConfiguration.builder()
.apiCallTimeout(Duration.ofHours(1))
.build())
.httpClientBuilder(UrlConnectionHttpClient.builder()
.connectionTimeout(Duration.ofHours(1))
.socketTimeout(Duration.ofHours(1)))
.credentialsProvider(credentialsProvider)
.build();
我的 spring 启动 applications.properties 请求超时设置：
spring.mvc.async.request-timeout=-1
我的 bootstrap class 上有 @EnableAsync。
请求通过 nginx 代理，位置设置为：
upstream myupstream {
server [SERVER-REDACTED]:[PORT-REDACTED];
keepalive 15;
}
server {
...
location ~ ^/(v1/bigsearch|actuator)/ {
proxy_pass http://myupstream;
proxy_http_version 1.1;
proxy_set_header Connection "Keep-Alive";
proxy_set_header Proxy-Connection "Keep-Alive";
client_max_body_size 2G;
}
...
}
以及相关的 nginx.conf 条目本身：
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 1d;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
proxy_read_timeout 1d;
proxy_send_timeout 1d;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
gzip on;
gzip_disable "msie6";
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
问题
我还没有做些什么来完成这项工作？
一如既往，我为一个问题苦苦挣扎了好几天，post 它在 SO 上，然后弄清楚。
即使错误表现为 SocketTimeoutException，问题实际上出在 nginx 代理缓冲上。
一段时间后，我觉得下载量总是超过 1G 很重要，直到我专门搜索它，我才在这里发现实际问题和解决方案：https://github.com/owncloud/client/issues/5706
将这些设置添加到 nginx.conf 文件中解决了它：
proxy_request_buffering off;
proxy_buffering off;