AWS SDK 无法执行 HTTP 请求以从 S3 获取元数据

Question

首先，我看到了与我类似的问题，但问题无处不在，即输入流未正确关闭。我的问题不一样。

压力测试中途发现错误（我们积极使用S3）

com.amazonaws.SdkClientException: Unable to execute HTTP request: Too many open files"
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1201)"
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1147)"
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796)"
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)"
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)"
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access0(AmazonHttpClient.java:698)"
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)"
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)"
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)"
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)"
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)"
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1486)"
at com.A.B.C.aws.s3.AwsS3Downloader.downloadObject(AwsS3Downloader.java:38)"
at com.A.B.C.aws.s3.AwsS3Util.getFileMetadata(AwsS3Util.java:1251)"

我第一次想到我有某个流需要关闭，但事实并非如此。我不在这里保存输入流。因此我 NOT 关闭 S3Object。

 AmazonS3 amazonS3 = getAmazonS3(awsClientConfig, needCrossAccntAccess, forceRefresh);
 S3Object object = AwsS3Downloader.downloadObject(amazonS3, bucketName, key);
 fileMetadata = object.getObjectMetadata().getRawMetadata();

也许在应用程序的某处我有未关闭的输入流，最终我在 http 调用期间出现错误？

注意：我不使用共享 AmazonS3 对象。

Answer 1

First time I was thinking that I have somewhere stream that needs to be closed but that's not the case. I don't hold the input stream here. Therefore I do NOT close the S3Object

你应该。

根据 the documentation:

The method is a simple getter and does not actually create a stream. If you retrieve an S3Object, you should close this input stream as soon as possible, because the object contents aren't buffered in memory and stream directly from Amazon S3. Further, failure to close this stream can cause the request pool to become blocked.

从这个描述中可能并不明显，但是这个调用对对象执行了 HTTP GET，其中包括对象内容。

如果您只关心对象元数据，则调用 getObjectMetadata()

Answer 2

我使用了几种工具来确定问题。

我创建了自定义指标来监控每个进程的文件描述符。我使用 AWS Cloud Watch Metrics 确定了哪个进程有问题。我创建了 shell 每 15 秒运行的脚本，并使用 AWS CLI 将数据点发送到 AWS Cloud Watch 指标。
我使用 YourKit 探查器找出内存泄漏的确切位置。

最终：

我增加了 ECS Fargate 环境的文件描述符大小，因为默认情况下 1024 是软限制，1024*4 是硬限制。这是一个漂亮的应用程序运行正在生产。
我确定了溪流未关闭的地方。（class 无法关闭并且 Java try-with-resources 无法正常工作。

AWS SDK 无法执行 HTTP 请求以从 S3 获取元数据

AWS SDK is unable to execute HTTP request to get metadata from S3

java

amazon-s3

amazon-web-services

aws-sdk