s3fs：如何使用 IMDS v2 在代理后面的 AWS EC2 实例上安装 S3 存储桶

Question

我们在将 AWS S3 存储桶（使用 s3fs v1.90）装载到 AWS EC2 实例时遇到了一些问题：

是运行Ubuntu18.04
需要 IMDS v2 会话令牌
在代理后面

curl 库返回的 HTTP 响应代码是 "417 - Expectation Failed"（下面有更多详细信息）。我在 www 上发现了一些提示，417 错误可能与我们的代理配置有关，请参阅：

https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z0000019LuWSAU

这让我相信我们的 NO_PROXY 配置没有被 s3fs 获取，但我真的不确定...

无论如何，这就是我们为安装存储桶而尝试做的事情：

sudo s3fs SOME_BUCKET ./mnt-s3/ -o iam_role=SOME_ROLE,url=https://s3.eu-central-1.amazonaws.com,endpoint=eu-central-1,allow_other,uid=1000,gid=1000,mp_umask=007,use_cache=/tmp/s3foldercache,dbglevel=debug -f

这是输出：


2021-09-08T12:36:27.681Z [INF] curl.cpp:CheckIAMCredentialUpdate(1826): IAM Access Token refreshing...
2021-09-08T12:36:27.681Z [INF]       curl.cpp:GetIAMCredentials(3068): [IAM role=SOME_ROLE]
2021-09-08T12:36:27.681Z [DBG] curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 31
2021-09-08T12:36:27.681Z [DBG] curl.cpp:RequestPerform(2509): connecting to URL http://169.254.169.254/latest/api/token
2021-09-08T12:36:27.682Z [ERR] curl.cpp:RequestPerform(2622): HTTP response code 417, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>417 - Expectation Failed</title>
 </head>
 <body>
  <h1>417 - Expectation Failed</h1>
 </body>
</html>
2021-09-08T12:36:27.682Z [ERR] curl.cpp:GetIAMCredentials(3105): AWS IMDSv2 token retrieval failed: -5
2021-09-08T12:36:27.682Z [DBG] curl.cpp:RequestPerform(2509): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/SOME_ROLE
2021-09-08T12:36:27.684Z [ERR] curl.cpp:RequestPerform(2622): HTTP response code 401, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>401 - Unauthorized</title>
 </head>
 <body>
  <h1>401 - Unauthorized</h1>
 </body>
</html>
2021-09-08T12:36:27.684Z [ERR] curl.cpp:CheckIAMCredentialUpdate(1830): IAM Access Token refresh failed
2021-09-08T12:36:27.684Z [DBG] curl_handlerpool.cpp:ReturnHandler(103): Return handler to pool
2021-09-08T12:36:27.684Z [INF] curl_handlerpool.cpp:ReturnHandler(110): Pool full: destroy the oldest handler
2021-09-08T12:36:27.685Z [CRT] s3fs.cpp:s3fs_check_service(3520): Failed to check IAM role name(SOME_ROLE).
2021-09-08T12:36:27.685Z [ERR] s3fs.cpp:s3fs_exit_fuseloop(3372): Exiting FUSE event loop due to errors

当运行直接卷曲时，我们确实收到了一个有效的 IMDS v2 令牌：

$ curl -v -X PUT -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" http://169.254.169.254/latest/api/token 

*   Trying 169.254.169.254...
* TCP_NODELAY set
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> PUT /latest/api/token HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.58.0
> Accept: */*
> X-aws-ec2-metadata-token-ttl-seconds: 21600
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Length: 56
< Content-Type: text/plain
< Date: Wed, 08 Sep 2021 13:14:02 GMT
< X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
< Connection: close
< Server: EC2ws
<
* Closing connection 0
SOME_TOKEN

最后，这是我们的代理配置（由环境变量定义）：

$ echo $HTTP_PROXY                                                                                                                                  
<SOME_HOST>:<SOME_PORT>

$ echo $NO_PROXY                                                                                                                                  
169.254.169.254,*.eu-central-1.amazonaws.com

所以，我最好的猜测是 s3fs 可能会忽略 NO_PROXY 变量，在向本地 IP 169.254.169.254 请求新令牌时尝试使用我们的代理。

Answer 1

正在 https://github.com/s3fs-fuse/s3fs-fuse/pull/1766

修复此问题

s3fs：如何使用 IMDS v2 在代理后面的 AWS EC2 实例上安装 S3 存储桶

s3fs: How to mount S3 buckets on AWS EC2 instances behind proxy, using IMDS v2

amazon-s3

amazon-ec2

s3fs