如何使用 cloudformation 将 TaskInstanceGroup 添加到 AWS EMR 以进行自动缩放？

Question

我想为任务节点添加一个自动缩放组，但无法让它与 cloudformation 一起工作。

同样的事情对 CoreInstanceGroup 也适用，如下所示。

Instances:
        CoreInstanceGroup:
          InstanceCount: 1
          InstanceType: !Ref CoreInstanceType
          Market: ON_DEMAND
          Name: Core Instance
          AutoScalingPolicy:
            Constraints:
              MinCapacity: !Ref CoreMinCapacity
              MaxCapacity: !Ref CoreMaxCapacity

当我用 TaskInstanceGroup 替换 CoreInstanceGroup 时，linter 发出警告，并且在运行脚本失败并显示错误属性未找到。

遇到了一个引用 TaskInstanceGroup 的 Terraform 脚本。有人有办法解决这个问题吗？

TIA。

Answer 1

任务实例组不属于 AWS::EMR::Cluster。这就是您收到错误的原因。
您已将 TaskInstanceGroup 作为不同的资源附加。
即 AWS::EMR::InstanceGroupConfig.

JobFlowId: !Ref myEMRCluster 这将确定它将在哪个集群中附加资源。 myEMRCluster是EMR的资源名称。
您可以附加多个具有不同自动缩放策略的 TaskInstanceGroup。
您还可以为您的任务组使用不同的 CloudFormation 脚本。在那种情况下，您必须像 JobFlowId: 'j-ABCD123456789'.

一样传递集群 ID

AWSTemplateFormatVersion: 2010-09-09
Resources:
  myEMRCluster:
    Type: 'AWS::EMR::Cluster'
    Properties: <... Your existing config ...>
  TaskInstanceGroup:
    Type: 'AWS::EMR::InstanceGroupConfig'
    Properties:
      InstanceRole: TASK
      InstanceCount: 0
      InstanceType: 'r5.8xlarge'
      Market: SPOT
      BidPrice: '1.110'
      Name: cfnTask
      JobFlowId: !Ref myEMRCluster
      AutoScalingPolicy:
        Constraints:
          MinCapacity: 0
          MaxCapacity: 40
        Rules:
          - Name: container-pending-ratio-scale-out
            Description: >-
              Replicates the default scale-out rule in the console for YARN
              memory.
            Action:
              SimpleScalingPolicyConfiguration:
                AdjustmentType: CHANGE_IN_CAPACITY
                ScalingAdjustment: 10
                CoolDown: 300
            Trigger:
              CloudWatchAlarmDefinition:
                ComparisonOperator: GREATER_THAN
                EvaluationPeriods: 2
                MetricName: ContainerPendingRatio
                Namespace: AWS/ElasticMapReduce
                Period: 300
                Threshold: 2
                Statistic: AVERAGE
                Unit: COUNT
                Dimensions:
                  - Key: JobFlowId
                    Value: '${emr.clusterId}'
          - Name: idle-scale-in
            Description: Replicates the default scale-in rule in the console for idle.
            Action:
              SimpleScalingPolicyConfiguration:
                AdjustmentType: CHANGE_IN_CAPACITY
                ScalingAdjustment: -40
                CoolDown: 300
            Trigger:
              CloudWatchAlarmDefinition:
                ComparisonOperator: LESS_THAN_OR_EQUAL
                EvaluationPeriods: 2
                MetricName: ContainerAllocated
                Namespace: AWS/ElasticMapReduce
                Period: 300
                Threshold: 0
                Statistic: AVERAGE
                Unit: COUNT
                Dimensions:
                  - Key: JobFlowId
                    Value: '${emr.clusterId}'
  myEMRStep:
    Type: 'AWS::EMR::Step'
    Properties: <... If you have any ...>

希望这对您有所帮助。

如何使用 cloudformation 将 TaskInstanceGroup 添加到 AWS EMR 以进行自动缩放？

How to Add TaskInstanceGroup to AWS EMR for autoscaling using cloudformation?

amazon-web-services

amazon-emr

autoscaling