AWS Batch:使用云形成将 efs 卷装载到容器中
AWS Batch: mount an efs volume into a container using cloud formation
我有一个带有作业定义的 AWS Batch 计算环境。
我使用 Cloud Formation 创建了这一切。
现在我想向此作业定义添加一个 EFS 卷(名称:EFS-000,文件系统 ID:fs-9999999)和一个装载点。
我读了
- https://docs.aws.amazon.com/AmazonECS/latest/developerguide/tutorial-efs-volumes.html
- https://aws.amazon.com/es/premiumsupport/knowledge-center/batch-mount-efs
在第一个 link 中,我们有一个任务定义示例(AWS ECS 而不是 AWS Batch 概念)
{
"containerDefinitions": [
{
"memory": 128,
"portMappings": [
{
"hostPort": 80,
"containerPort": 80,
"protocol": "tcp"
}
],
"essential": true,
"mountPoints": [
{
"containerPath": "/usr/share/nginx/html",
"sourceVolume": "efs-html"
}
],
"name": "nginx",
"image": "nginx"
}
],
"volumes": [
{
"name": "efs-html",
"efsVolumeConfiguration": {
"fileSystemId": "fs-1324abcd",
"transitEncryption": "ENABLED"
}
}
],
"family": "efs-tutorial"
}
似乎很容易将正确的代码添加到我的 Cloud Formation 配方中(我选择了 yaml 语法)。在我的 ContainerDefinition 中,我添加了...
Volumes:
- Name: SRV
EfsVolumeConfiguration:
FileSystemId: fs-9999999
TransitEncryption: ENABLED
但是当我 运行 我得到了 Cloud Formation 配方....
The following resource(s) failed to update: [ContentJob1].
Property validation failure: [Encountered unsupported properties in {/ContainerProperties/Volumes/0}: [EfsVolumeConfiguration]]
如果 EfsVolumeConfiguration 不是有效的 属性...
如何使用 Cloud Formation 将 EFS 卷添加到 AWS Batch 作业定义?
AWS 批处理 added EFS Volume support on April 1, 2021.
用户文档位于 https://docs.aws.amazon.com/batch/latest/userguide/efs-volumes.html
可在 https://aws.amazon.com/blogs/hpc/introducing-support-for-per-job-amazon-efs-volumes-in-aws-batch/
访问博客 post
在 CloudFormation 支持 EFS 之前,我一直使用这个解决方案。不过还是有用的。
我创建启动模板
LaunchTemplate:
Type: AWS::EC2::LaunchTemplate
Properties:
LaunchTemplateName: !Join ['', [!Ref ServiceName, '-EC2-', LaunchTemplate]]
LaunchTemplateData:
UserData:
Fn::Base64: !Sub |
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="
--==MYBOUNDARY==
Content-Type: text/cloud-config; charset="us-ascii"
runcmd:
- mkdir /mnt/efs-misc
- mkdir /mnt/efs-jobs
- echo "${EfsJobs}:/ /mnt/efs-jobs nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=300,retrans=2,noresvport 0 0" >> /etc/fstab
- echo "${EfsMisc}:/ /mnt/efs-misc nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=300,retrans=2,noresvport 0 0" >> /etc/fstab
- mount -a
--==MYBOUNDARY==--
InstanceMarketOptions:
MarketType: spot
SpotOptions:
MaxPrice: 0.096
然后我创建了一个使用此模板的计算环境。
BatchCompute:
Type: 'AWS::Batch::ComputeEnvironment'
DependsOn: LaunchTemplate
Properties:
ComputeEnvironmentName: !Join ['', [!Ref ServiceName, '-EC2']]
ComputeResources:
AllocationStrategy: SPOT_CAPACITY_OPTIMIZED
DesiredvCpus: 0
Ec2KeyPair: !Join ['-', [MyComputeEnv, !Ref EnvironmentLow]]
InstanceRole: !GetAtt BatchInstanceProfile.Arn
InstanceTypes:
- c4.large
- c5.large
LaunchTemplate:
LaunchTemplateId: !Ref LaunchTemplate
Version: $Latest
MaxvCpus: 256
MinvCpus: 0
DesiredvCpus: 0
SecurityGroupIds:
- !Ref DefaultSecurityGroup
Subnets: !Split [',', !Join [',', [!Ref SubnetA, !Ref SubnetB, !Ref SubnetC ]]]
Type: SPOT
SpotIamFleetRole: SpotFleetTagServiceRole
ServiceRole: !Ref BatchServiceRole
State: ENABLED
Type: Managed
最后,我使用 AWS::Batch::JobDefinition 的 ContainerProperties/[Volumes|MountPoints] 添加我的 EFS 卷。
ContentJob0:
Type: 'AWS::Batch::JobDefinition'
Properties:
Type: Container
ContainerProperties:
Command:
- ls
Environment:
- Name: AAA
Value: AAA
ExecutionRoleArn: !GetAtt BatchJobRole.Arn
Image: !Join ['', [!Ref 'AWS::AccountId','.dkr.ecr.', !Ref 'AWS::Region', '.amazonaws.com/', !Ref Image ]]
LogConfiguration:
LogDriver: awslogs
JobRoleArn: !Ref BatchContainerRole
Memory: 2048
Vcpus: 2
Volumes:
- Name: MISC
Host:
SourcePath: '/mnt/efs-0'
- Name: JOBS
Host:
SourcePath: '/mnt/efs-1'
MountPoints:
- SourceVolume: MISC
ContainerPath: '/mnt/efs-0'
- SourceVolume: JOBS
ContainerPath: '/mnt/efs-1'
JobDefinitionName: !Join ['-', ['MyJob', !Ref ServiceName, 'EC2', '0']]
PlatformCapabilities:
- EC2
PropagateTags: true
RetryStrategy:
Attempts: 3
Timeout:
AttemptDurationSeconds: 300
因为我需要在 LaunchTemplate 中设置 SPOT 实例的价格和其他东西,这对我来说是一个很好的解决方案。
InstanceMarketOptions:
MarketType: spot
SpotOptions:
MaxPrice: 0.096
我有一个带有作业定义的 AWS Batch 计算环境。
我使用 Cloud Formation 创建了这一切。
现在我想向此作业定义添加一个 EFS 卷(名称:EFS-000,文件系统 ID:fs-9999999)和一个装载点。
我读了
- https://docs.aws.amazon.com/AmazonECS/latest/developerguide/tutorial-efs-volumes.html
- https://aws.amazon.com/es/premiumsupport/knowledge-center/batch-mount-efs
在第一个 link 中,我们有一个任务定义示例(AWS ECS 而不是 AWS Batch 概念)
{
"containerDefinitions": [
{
"memory": 128,
"portMappings": [
{
"hostPort": 80,
"containerPort": 80,
"protocol": "tcp"
}
],
"essential": true,
"mountPoints": [
{
"containerPath": "/usr/share/nginx/html",
"sourceVolume": "efs-html"
}
],
"name": "nginx",
"image": "nginx"
}
],
"volumes": [
{
"name": "efs-html",
"efsVolumeConfiguration": {
"fileSystemId": "fs-1324abcd",
"transitEncryption": "ENABLED"
}
}
],
"family": "efs-tutorial"
}
似乎很容易将正确的代码添加到我的 Cloud Formation 配方中(我选择了 yaml 语法)。在我的 ContainerDefinition 中,我添加了...
Volumes:
- Name: SRV
EfsVolumeConfiguration:
FileSystemId: fs-9999999
TransitEncryption: ENABLED
但是当我 运行 我得到了 Cloud Formation 配方....
The following resource(s) failed to update: [ContentJob1].
Property validation failure: [Encountered unsupported properties in {/ContainerProperties/Volumes/0}: [EfsVolumeConfiguration]]
如果 EfsVolumeConfiguration 不是有效的 属性...
如何使用 Cloud Formation 将 EFS 卷添加到 AWS Batch 作业定义?
AWS 批处理 added EFS Volume support on April 1, 2021.
用户文档位于 https://docs.aws.amazon.com/batch/latest/userguide/efs-volumes.html
可在 https://aws.amazon.com/blogs/hpc/introducing-support-for-per-job-amazon-efs-volumes-in-aws-batch/
访问博客 post在 CloudFormation 支持 EFS 之前,我一直使用这个解决方案。不过还是有用的。
我创建启动模板
LaunchTemplate:
Type: AWS::EC2::LaunchTemplate
Properties:
LaunchTemplateName: !Join ['', [!Ref ServiceName, '-EC2-', LaunchTemplate]]
LaunchTemplateData:
UserData:
Fn::Base64: !Sub |
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="
--==MYBOUNDARY==
Content-Type: text/cloud-config; charset="us-ascii"
runcmd:
- mkdir /mnt/efs-misc
- mkdir /mnt/efs-jobs
- echo "${EfsJobs}:/ /mnt/efs-jobs nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=300,retrans=2,noresvport 0 0" >> /etc/fstab
- echo "${EfsMisc}:/ /mnt/efs-misc nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=300,retrans=2,noresvport 0 0" >> /etc/fstab
- mount -a
--==MYBOUNDARY==--
InstanceMarketOptions:
MarketType: spot
SpotOptions:
MaxPrice: 0.096
然后我创建了一个使用此模板的计算环境。
BatchCompute:
Type: 'AWS::Batch::ComputeEnvironment'
DependsOn: LaunchTemplate
Properties:
ComputeEnvironmentName: !Join ['', [!Ref ServiceName, '-EC2']]
ComputeResources:
AllocationStrategy: SPOT_CAPACITY_OPTIMIZED
DesiredvCpus: 0
Ec2KeyPair: !Join ['-', [MyComputeEnv, !Ref EnvironmentLow]]
InstanceRole: !GetAtt BatchInstanceProfile.Arn
InstanceTypes:
- c4.large
- c5.large
LaunchTemplate:
LaunchTemplateId: !Ref LaunchTemplate
Version: $Latest
MaxvCpus: 256
MinvCpus: 0
DesiredvCpus: 0
SecurityGroupIds:
- !Ref DefaultSecurityGroup
Subnets: !Split [',', !Join [',', [!Ref SubnetA, !Ref SubnetB, !Ref SubnetC ]]]
Type: SPOT
SpotIamFleetRole: SpotFleetTagServiceRole
ServiceRole: !Ref BatchServiceRole
State: ENABLED
Type: Managed
最后,我使用 AWS::Batch::JobDefinition 的 ContainerProperties/[Volumes|MountPoints] 添加我的 EFS 卷。
ContentJob0:
Type: 'AWS::Batch::JobDefinition'
Properties:
Type: Container
ContainerProperties:
Command:
- ls
Environment:
- Name: AAA
Value: AAA
ExecutionRoleArn: !GetAtt BatchJobRole.Arn
Image: !Join ['', [!Ref 'AWS::AccountId','.dkr.ecr.', !Ref 'AWS::Region', '.amazonaws.com/', !Ref Image ]]
LogConfiguration:
LogDriver: awslogs
JobRoleArn: !Ref BatchContainerRole
Memory: 2048
Vcpus: 2
Volumes:
- Name: MISC
Host:
SourcePath: '/mnt/efs-0'
- Name: JOBS
Host:
SourcePath: '/mnt/efs-1'
MountPoints:
- SourceVolume: MISC
ContainerPath: '/mnt/efs-0'
- SourceVolume: JOBS
ContainerPath: '/mnt/efs-1'
JobDefinitionName: !Join ['-', ['MyJob', !Ref ServiceName, 'EC2', '0']]
PlatformCapabilities:
- EC2
PropagateTags: true
RetryStrategy:
Attempts: 3
Timeout:
AttemptDurationSeconds: 300
因为我需要在 LaunchTemplate 中设置 SPOT 实例的价格和其他东西,这对我来说是一个很好的解决方案。
InstanceMarketOptions:
MarketType: spot
SpotOptions:
MaxPrice: 0.096