如何解决 Argo 输出参数大小限制？

Question

我有一个在 JSON 数组上循环的 Argo 工作流。当列表变得太大时，我会收到这样的错误：

time="some-time" level=fatal msg="Pod \"some-pod-name\" is invalid: metadata.annotations: Too long: must have at most 262144 characters"

或者，in newer versions of Argo：

Output is larger than the maximum allowed size of 256 kB, only the last 256 kB were saved

如何在不达到大小限制的情况下遍历这个大 JSON 数组？

我的工作流程看起来有点像这样，但是有一个更大的 JSON 数组：

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: loops-sequence-
spec:
  entrypoint: loops-sequence
  templates:
  - name: loops-sequence
    steps:
    - - name: get-items
        template: get-items
    - - name: sequence-param
        template: echo
        arguments:
          parameters:
          - name: item
            value: "{{item}}"
        withParam: "{{steps.get-items.outputs.parameters.items}}"
  - name: get-items
    container:
      image: alpine:latest
      command: ["/bin/sh", "-c"]
      args: ["echo '[\"a\", \"b\", \"c\"]' > /tmp/items"]
    outputs:
      parameters:
      - name: items
        valueFrom:
          path: /tmp/items
  - name: echo
    inputs:
      parameters:
      - name: item
    container:
      image: stedolan/jq:latest
      command: [echo, "{{inputs.parameters.item}}"]

Answer 1

不是将 JSON 数组写入输出参数，而是将其写入工件并将其长度写入输出参数。然后你可以使用 withSequence 遍历数组索引并从 JSON 工件中检索相应的项目。

这是一个硬编码 JSON 数组和项目计数的示例。您的 get-items 步骤肯定会更复杂。

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: loops-sequence-
spec:
  entrypoint: loops-sequence
  templates:
  - name: loops-sequence
    steps:
    - - name: get-items
        template: get-items
    - - name: sequence-param
        template: echo
        arguments:
          parameters:
          - name: index
            value: "{{item}}"
          artifacts:
          - name: items
            from: "{{steps.get-items.outputs.artifacts.items}}"
        withSequence:
          count: "{{steps.get-items.outputs.parameters.count}}"
  - name: get-items
    container:
      image: alpine:latest
      command: ["/bin/sh", "-c"]
      args: ["echo '[\"a\", \"b\", \"c\"]' > /tmp/items && echo '3' > /tmp/count"]
    outputs:
      artifacts:
      - name: items
        path: /tmp/items
      parameters:
      - name: count
        valueFrom:
          path: /tmp/count
  - name: echo
    inputs:
      parameters:
      - name: index
      artifacts:
      - name: items
        path: /tmp/items
    container:
      image: stedolan/jq:latest
      command: [sh, -c]
      args: ["cat /tmp/items | jq '.[{{inputs.parameters.index}}]'"]

如何解决 Argo 输出参数大小限制？

How can I work around Argo output parameter size limit?

scalability

argo-workflows