如何获取 aws-iam-token 以使用 IRSA 访问 S3?
How do you obtain an aws-iam-token to access S3 using IRSA?
我在 Terraform 中创建了一个 IRSA 角色,以便 K8s 作业可以使用关联的服务帐户来访问 S3 存储桶,但我在作业中不断收到 AccessDenied
错误。
我首先在我们的 eks
模块中使用 enable_irsa = true
在我们的 EKS 集群中启用了 IRSA。
然后我创建了一个简单的 aws_iam_policy
作为:
resource "aws_iam_policy" "eks_s3_access_policy" {
name = "eks_s3_access_policy"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = [
"s3:*",
]
Effect = "Allow"
Resource = "arn:aws:s3:::*"
},
]
})
}
和一个iam-assumable-role-with-oidc
:
module "iam_assumable_role_with_oidc_for_s3_access" {
source = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
version = "~> 3.0"
create_role = true
role_name = "eks-s3-access"
role_description = "Role to access s3 bucket"
tags = { Role = "eks_s3_access_policy" }
provider_url = replace(module.eks.cluster_oidc_issuer_url, "https://", "")
role_policy_arns = [aws_iam_policy.eks_s3_access_policy.arn]
number_of_role_policy_arns = 1
oidc_fully_qualified_subjects = ["system:serviceaccount:default:my-user"]
}
我使用 Helm 创建了一个 K8s 服务帐户,例如:
Name: my-user
Namespace: default
Labels: app.kubernetes.io/managed-by=Helm
Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::111111:role/eks-s3-access
meta.helm.sh/release-name: XXXX
meta.helm.sh/release-namespace: default
Image pull secrets: <none>
Mountable secrets: my-user-token-kwwpq
Tokens: my-user-token-kwwpq
Events: <none>
最后,使用 K8s API 从作业模板创建作业:
apiVersion: batch/v1
kind: Job
metadata:
name: job
namespace: default
spec:
template:
spec:
serviceAccountName: my-user
containers:
- name: {{ .Chart.Name }}
env:
- name: AWS_ROLE_ARN
value: arn:aws:iam::746181457053:role/eks-s3-access
- name: AWS_WEB_IDENTITY_TOKEN_FILE
value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
volumeMounts:
- mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
name: aws-iam-token
readOnly: true
volumes:
- name: aws-iam-token
projected:
defaultMode: 420
sources:
- serviceAccountToken:
audience: sts.amazonaws.com
expirationSeconds: 86400
path: token
当作业尝试获取指定的凭据时,指定的令牌不存在:
2021-08-03 18:02:41 Refreshing temporary credentials failed during mandatory refresh period.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 291, in _protected_refresh
metadata = await self._refresh_using()
File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 345, in fetch_credentials
return await self._get_cached_credentials()
File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 355, in _get_cached_credentials
response = await self._get_credentials()
File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 410, in _get_credentials
kwargs = self._assume_role_kwargs()
File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 420, in _assume_role_kwargs
identity_token = self._web_identity_token_loader()
File "/usr/local/lib/python3.7/site-packages/botocore/utils.py", line 2365, in __call__
with self._open(self._web_identity_token_path) as token_file:
FileNotFoundError: [Errno 2] No such file or directory: '/var/run/secrets/eks.amazonaws.com/serviceaccount/token'
根据 https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/ 中的描述,webhook 通常会在创建 pod 时创建这些凭据。然而,由于我们在 k8s 集群中按需创建新的 k8s 作业,我怀疑 webhook 没有创建任何此类凭据。
如何请求在 K8s 集群中创建正确的凭据?有没有办法从集群中实例化 webhook?
有几件事可能会导致失败。
- 检查 IRSA 角色的所有设置。对于信任关系设置,请检查命名空间的名称和服务帐户的名称是否正确。只有这些设置匹配角色才能承担。
- 当 pod 处于 运行 时尝试使用 shell 访问 pod。检查“AWS_*”环境变量的内容。检查 AWS_ROLE_ARN 指向正确的角色。检查 AWS_WEB_IDENTITY_TOKEN_FILE 指向的文件是否在其位置并且可读。只需尝试对文件执行
cat
以查看它是否可读。
- 如果您 运行 您的 Pod 是非根用户(出于安全原因推荐这样做),请确保 运行 Pod 的用户可以访问该文件。如果不是,请调整 pod 的 securityContext。也许
fsGroup
的设置在这里可以提供帮助。 https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#security-context
- 确保您的 pos 使用的 SDK 支持 IRSA。如果您使用的是较旧的 SDK,则可能不支持 IRSA。查看 IRSA 文档以了解受支持的 SDK 版本。 https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
我在 Terraform 中创建了一个 IRSA 角色,以便 K8s 作业可以使用关联的服务帐户来访问 S3 存储桶,但我在作业中不断收到 AccessDenied
错误。
我首先在我们的 eks
模块中使用 enable_irsa = true
在我们的 EKS 集群中启用了 IRSA。
然后我创建了一个简单的 aws_iam_policy
作为:
resource "aws_iam_policy" "eks_s3_access_policy" {
name = "eks_s3_access_policy"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = [
"s3:*",
]
Effect = "Allow"
Resource = "arn:aws:s3:::*"
},
]
})
}
和一个iam-assumable-role-with-oidc
:
module "iam_assumable_role_with_oidc_for_s3_access" {
source = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
version = "~> 3.0"
create_role = true
role_name = "eks-s3-access"
role_description = "Role to access s3 bucket"
tags = { Role = "eks_s3_access_policy" }
provider_url = replace(module.eks.cluster_oidc_issuer_url, "https://", "")
role_policy_arns = [aws_iam_policy.eks_s3_access_policy.arn]
number_of_role_policy_arns = 1
oidc_fully_qualified_subjects = ["system:serviceaccount:default:my-user"]
}
我使用 Helm 创建了一个 K8s 服务帐户,例如:
Name: my-user
Namespace: default
Labels: app.kubernetes.io/managed-by=Helm
Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::111111:role/eks-s3-access
meta.helm.sh/release-name: XXXX
meta.helm.sh/release-namespace: default
Image pull secrets: <none>
Mountable secrets: my-user-token-kwwpq
Tokens: my-user-token-kwwpq
Events: <none>
最后,使用 K8s API 从作业模板创建作业:
apiVersion: batch/v1
kind: Job
metadata:
name: job
namespace: default
spec:
template:
spec:
serviceAccountName: my-user
containers:
- name: {{ .Chart.Name }}
env:
- name: AWS_ROLE_ARN
value: arn:aws:iam::746181457053:role/eks-s3-access
- name: AWS_WEB_IDENTITY_TOKEN_FILE
value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
volumeMounts:
- mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
name: aws-iam-token
readOnly: true
volumes:
- name: aws-iam-token
projected:
defaultMode: 420
sources:
- serviceAccountToken:
audience: sts.amazonaws.com
expirationSeconds: 86400
path: token
当作业尝试获取指定的凭据时,指定的令牌不存在:
2021-08-03 18:02:41 Refreshing temporary credentials failed during mandatory refresh period.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 291, in _protected_refresh
metadata = await self._refresh_using()
File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 345, in fetch_credentials
return await self._get_cached_credentials()
File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 355, in _get_cached_credentials
response = await self._get_credentials()
File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 410, in _get_credentials
kwargs = self._assume_role_kwargs()
File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 420, in _assume_role_kwargs
identity_token = self._web_identity_token_loader()
File "/usr/local/lib/python3.7/site-packages/botocore/utils.py", line 2365, in __call__
with self._open(self._web_identity_token_path) as token_file:
FileNotFoundError: [Errno 2] No such file or directory: '/var/run/secrets/eks.amazonaws.com/serviceaccount/token'
根据 https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/ 中的描述,webhook 通常会在创建 pod 时创建这些凭据。然而,由于我们在 k8s 集群中按需创建新的 k8s 作业,我怀疑 webhook 没有创建任何此类凭据。
如何请求在 K8s 集群中创建正确的凭据?有没有办法从集群中实例化 webhook?
有几件事可能会导致失败。
- 检查 IRSA 角色的所有设置。对于信任关系设置,请检查命名空间的名称和服务帐户的名称是否正确。只有这些设置匹配角色才能承担。
- 当 pod 处于 运行 时尝试使用 shell 访问 pod。检查“AWS_*”环境变量的内容。检查 AWS_ROLE_ARN 指向正确的角色。检查 AWS_WEB_IDENTITY_TOKEN_FILE 指向的文件是否在其位置并且可读。只需尝试对文件执行
cat
以查看它是否可读。 - 如果您 运行 您的 Pod 是非根用户(出于安全原因推荐这样做),请确保 运行 Pod 的用户可以访问该文件。如果不是,请调整 pod 的 securityContext。也许
fsGroup
的设置在这里可以提供帮助。 https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#security-context - 确保您的 pos 使用的 SDK 支持 IRSA。如果您使用的是较旧的 SDK,则可能不支持 IRSA。查看 IRSA 文档以了解受支持的 SDK 版本。 https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html