修改现有的 yaml 文件并添加新的数据和注释

Modify existing yaml file and add new data and comments

最近看到go yaml lib有新版本(V3)

具有 nodes 功能(在我看来这是一个杀手级功能 :)),它可以在不改变文件结构的情况下帮助修改 yamls

但由于它是相当新的(从上周开始),我没有找到一些有用的文档和示例来说明我需要的上下文(添加新的object/node并保持文件结构不变而不删除注释等)

我需要的是操作yaml文件

例如

假设我有这个 yaml 文件

version: 1
type: verbose
kind : bfr

# my list of applications
applications:
  - name: app1
    kind: nodejs
    path: app1
    exec:
      platforms: k8s
      builder: test

现在我得到了一个 json 对象(例如 app2),我需要将其插入现有文件

[

    {
        "comment: "Second app",
        "name": "app2",
        "kind": "golang",
        "path": "app2",
        "exec": {
            "platforms": "dockerh",
            "builder": "test"
        }
    }
]

并且我需要在第一次申请之后将其添加到 yml 文件中,(applications 是应用程序的数组)

version: 1
type: verbose
kind : bfr

# my list of applications
applications:

#  First app
  - name: app1
    kind: nodejs
    path: app1
    exec:
      platforms: k8s
      builder: test

# Second app
  - name: app2
    kind: golang
    path: app2
    exec:
      platforms: dockerh
      builder: test

是否可以从 yaml 文件添加新的 json 对象?同时删除现有的

我也找到了这个博客 https://blog.ubuntu.com/2019/04/05/api-v3-of-the-yaml-package-for-go-is-available

这是表示对象的类型

type VTS struct {
    version string       `yaml:"version"`
    types   string       `yaml:"type"`
    kind    string       `yaml:"kind,omitempty"`
    apps    Applications `yaml:"applications,omitempty"`
}

type Applications []struct {
    Name string `yaml:"name,omitempty"`
    Kind string `yaml:"kind,omitempty"`
    Path string `yaml:"path,omitempty"`
    Exec struct {
        Platforms string `yaml:"platforms,omitempty"`
        Builder   string `yaml:"builder,omitempty"`
    } `yaml:"exec,omitempty"`
}

更新

在测试 wiil7200 提供的解决方案后,我发现了 2 个问题

我最后用写到文件 err = ioutil.WriteFile("output.yaml", b, 0644)

并且 yaml 输出有 2 个问题。

  1. The array of the application is starting from the comments, it should start from the name

  2. After the name entry the kind property and all others after are not aligned to the name

知道如何解决这些问题吗?关于 comments 问题,假设我是从其他 属性 那里得到的 而不是来自 json(如果它使它更简单)

version: 1
type: verbose
kind: bfr


# my list of applications
applications:
-   #  First app
name: app1
    kind: nodejs
    path: app1
    exec:
        platforms: k8s
        builder: test
-   # test 1
name: app2
    kind: golang
    path: app2
    exec:
        platform: dockerh
        builder: test

首先,让我先说使用 yaml.Node 从有效的 yaml 编组时不会生成有效的 yaml,如下例所示。可能应该提出问题。

package main

import (
    "fmt"
    "log"

    "gopkg.in/yaml.v3"
)

var (
    sourceYaml = `version: 1
type: verbose
kind : bfr

# my list of applications
applications:

#  First app
  - name: app1
    kind: nodejs
    path: app1
    exec:
      platforms: k8s
      builder: test
`
)

func main() {
    t := yaml.Node{}

    err := yaml.Unmarshal([]byte(sourceYaml), &t)
    if err != nil {
        log.Fatalf("error: %v", err)
    }

    b, err := yaml.Marshal(&t)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(string(b))
}

在 go 版本 go1.12.3 中生成以下无效的 yaml windows/amd64

version: 1
type: verbose
kind: bfr


# my list of applications
applications:
-   #  First app
name: app1
    kind: nodejs
    path: app1
    exec:
        platforms: k8s
        builder: test

其次,使用

这样的结构
type VTS struct {
    Version string       `yaml:"version" json:"version"`
    Types   string       `yaml:"type" json:"type"`
    Kind    string       `yaml:"kind,omitempty" json:"kind,omitempty"`
    Apps    yaml.Node `yaml:"applications,omitempty" json:"applications,omitempty"`
}

从 ubuntu 的博客和源文档来看,它似乎可以正确识别结构中作为节点的字段并单独构建该树,但事实并非如此。 解组时,它将提供正确的节点树,但重新编组时,它将生成以下 yaml,其中包含 yaml.Node 公开的所有字段。遗憾的是我们不能走这条路,必须另辟蹊径。

version: "1"
type: verbose
kind: bfr
applications:
    kind: 2
    style: 0
    tag: '!!seq'
    value: ""
    anchor: ""
    alias: null
    content:
    -   #  First app
name: app1
        kind: nodejs
        path: app1
        exec:
            platforms: k8s
            builder: test
    headcomment: ""
    linecomment: ""
    footcomment: ""
    line: 9
    column: 3

忽略结构中 yaml.Node 的第一个问题和编组错误(在 gopkg.in/yaml.v3 v3.0.0-20190409140830-cdc409dda467 上)我们现在可以开始操作节点包公开。不幸的是,没有可以轻松添加节点的抽象,因此用途可能会有所不同,并且识别节点可能会很痛苦。反思可能会有所帮助,所以我把它留给你作为练习。

您会发现注释 spew.Dumps 以良好的格式转储整个节点树,这有助于在将节点添加到源树时进行调试。

您当然也可以删除节点,您只需要确定需要删除哪些特定节点即可。如果它是映射或序列,您只需确保删除父节点。

package main

import (
    "encoding/json"
    "fmt"
    "log"

    "gopkg.in/yaml.v3"
)

var (
    sourceYaml = `version: 1
type: verbose
kind : bfr

# my list of applications
applications:

#  First app
  - name: app1
    kind: nodejs
    path: app1
    exec:
      platforms: k8s
      builder: test
`
    modifyJsonSource = `
[

    {
        "comment": "Second app",
        "name": "app2",
        "kind": "golang",
        "path": "app2",
        "exec": {
            "platforms": "dockerh",
            "builder": "test"
        }
    }
]
`
)

// VTS Need to Make Fields Public otherwise unmarshalling will not fill in the unexported fields.
type VTS struct {
    Version string       `yaml:"version" json:"version"`
    Types   string       `yaml:"type" json:"type"`
    Kind    string       `yaml:"kind,omitempty" json:"kind,omitempty"`
    Apps    Applications `yaml:"applications,omitempty" json:"applications,omitempty"`
}

type Applications []struct {
    Name string `yaml:"name,omitempty" json:"name,omitempty"`
    Kind string `yaml:"kind,omitempty" json:"kind,omitempty"`
    Path string `yaml:"path,omitempty" json:"path,omitempty"`
    Exec struct {
        Platforms string `yaml:"platforms,omitempty" json:"platforms,omitempty"`
        Builder   string `yaml:"builder,omitempty" json:"builder,omitempty"`
    } `yaml:"exec,omitempty" json:"exec,omitempty"`
    Comment string `yaml:"comment,omitempty" json:"comment,omitempty"`
}

func main() {
    t := yaml.Node{}

    err := yaml.Unmarshal([]byte(sourceYaml), &t)
    if err != nil {
        log.Fatalf("error: %v", err)
    }

    // Look for the Map Node with the seq array of items
    applicationNode := iterateNode(&t, "applications")

    // spew.Dump(iterateNode(&t, "applications"))

    var addFromJson Applications
    err = json.Unmarshal([]byte(modifyJsonSource), &addFromJson)
    if err != nil {
        log.Fatalf("error: %v", err)
    }

    // Delete the Original Applications the following options:
    // applicationNode.Content = []*yaml.Node{}
    // deleteAllContents(applicationNode)
    deleteApplication(applicationNode, "name", "app1")


    for _, app := range addFromJson {
        // Build New Map Node for new sequences coming in from json
        mapNode := &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"}

        // Build Name, Kind, and Path Nodes
        mapNode.Content = append(mapNode.Content, buildStringNodes("name", app.Name, app.Comment)...)
        mapNode.Content = append(mapNode.Content, buildStringNodes("kind", app.Kind, "")...)
        mapNode.Content = append(mapNode.Content, buildStringNodes("path", app.Path, "")...)

        // Build the Exec Nodes and the Platform and Builder Nodes within it
        keyMapNode, keyMapValuesNode := buildMapNodes("exec")
        keyMapValuesNode.Content = append(keyMapValuesNode.Content, buildStringNodes("platform", app.Exec.Platforms, "")...)
        keyMapValuesNode.Content = append(keyMapValuesNode.Content, buildStringNodes("builder", app.Exec.Builder, "")...)

        // Add to parent map Node
        mapNode.Content = append(mapNode.Content, keyMapNode, keyMapValuesNode)

        // Add to applications Node
        applicationNode.Content = append(applicationNode.Content, mapNode)
    }
    // spew.Dump(t)
    b, err := yaml.Marshal(&t)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(string(b))
}

// iterateNode will recursive look for the node following the identifier Node,
// as go-yaml has a node for the key and the value itself
// we want to manipulate the value Node
func iterateNode(node *yaml.Node, identifier string) *yaml.Node {
    returnNode := false
    for _, n := range node.Content {
        if n.Value == identifier {
            returnNode = true
            continue
        }
        if returnNode {
            return n
        }
        if len(n.Content) > 0 {
            ac_node := iterateNode(n, identifier)
            if ac_node != nil {
                return ac_node
            }
        }
    }
    return nil
}

// deleteAllContents will remove all the contents of a node
// Mark sure to pass the correct node in otherwise bad things will happen
func deleteAllContents(node *yaml.Node) {
    node.Content = []*yaml.Node{}
}

// deleteApplication expects that a sequence Node with all the applications are present
// if the key value are not found it will not log any errors, and return silently
// this is expecting a map like structure for the applications
func deleteApplication(node *yaml.Node, key, value string) {
    state := -1
    indexRemove := -1
    for index, parentNode := range node.Content {
        for _, childNode := range parentNode.Content {
            if key == childNode.Value && state == -1 {
                state += 1
                continue // found expected move onto next
            }
            if value == childNode.Value && state == 0 {
                state += 1
                indexRemove = index
                break // found the target exit out of the loop
            } else if state == 0 {
                state = -1
            }
        }
    }
    if state == 1 {
        // Remove node from contents
        // node.Content = append(node.Content[:indexRemove], node.Content[indexRemove+1:]...)
        // Don't Do this you might have a potential memory leak source: https://github.com/golang/go/wiki/SliceTricks
        // Since the underlying nodes are pointers
        length := len(node.Content)
        copy(node.Content[indexRemove:], node.Content[indexRemove+1:])
        node.Content[length-1] = nil
        node.Content = node.Content[:length-1]
    }
}


// buildStringNodes builds Nodes for a single key: value instance
func buildStringNodes(key, value, comment string) []*yaml.Node {
    keyNode := &yaml.Node{
        Kind:        yaml.ScalarNode,
        Tag:         "!!str",
        Value:       key,
        HeadComment: comment,
    }
    valueNode := &yaml.Node{
        Kind:  yaml.ScalarNode,
        Tag:   "!!str",
        Value: value,
    }
    return []*yaml.Node{keyNode, valueNode}
}

// buildMapNodes builds Nodes for a key: map instance
func buildMapNodes(key string) (*yaml.Node, *yaml.Node) {
    n1, n2 := &yaml.Node{
        Kind:  yaml.ScalarNode,
        Tag:   "!!str",
        Value: key,
    }, &yaml.Node{Kind: yaml.MappingNode,
        Tag: "!!map",
    }
    return n1, n2
}

生成 yaml

version: 1
type: verbose
kind: bfr


# my list of applications
applications:
-   #  First app
name: app1
    kind: nodejs
    path: app1
    exec:
        platforms: k8s
        builder: test
-   # Second app
name: app2
    kind: golang
    path: app2
    exec:
        platform: dockerh
        builder: test

您可以创建一个新节点并直接追加到内容中,而无需删除之前的节点。下面的例子说明了这一点:

package main

import (
    "fmt"
    "log"

    "gopkg.in/yaml.v3"
)

var (
    sourceYaml = `version: 1
type: verbose
kind : bfr

# my list of applications
applications:

#  First app
  - name: app1
    kind: nodejs
    path: app1
    exec:
      platforms: k8s
      builder: test
`
)

type Application struct {
    Name string `yaml:"name,omitempty" json:"name,omitempty"`
    Kind string `yaml:"kind,omitempty" json:"kind,omitempty"`
    Path string `yaml:"path,omitempty" json:"path,omitempty"`
    Exec struct {
        Platforms string `yaml:"platforms,omitempty" json:"platforms,omitempty"`
        Builder   string `yaml:"builder,omitempty" json:"builder,omitempty"`
    } `yaml:"exec,omitempty" json:"exec,omitempty"`
}

func newApplicationNode(
    name string,
    kind string,
    path string,
    platforms string,
    builder string,
    comment string) (*yaml.Node, error) {

    app := Application{
        Name: name,
        Kind: kind,
        Path: path,
        Exec: struct {
            Platforms string `yaml:"platforms,omitempty" json:"platforms,omitempty"`
            Builder   string `yaml:"builder,omitempty" json:"builder,omitempty"`
        }{platforms, builder},
    }
    marshalledApp, err := yaml.Marshal(&app)
    if err != nil {
        return nil, err
    }

    node := yaml.Node{}
    if err := yaml.Unmarshal(marshalledApp, &node); err != nil {
        return nil, err
    }
    node.Content[0].HeadComment = comment
    return &node, nil
}

func main() {
    yamlNode := yaml.Node{}

    err := yaml.Unmarshal([]byte(sourceYaml), &yamlNode)
    if err != nil {
        log.Fatalf("error: %v", err)
    }

    newApp, err := newApplicationNode("app2", "golang", "app2", "dockerh",
        "test", "Second app")
    if err != nil {
        log.Fatalf("error: %v", err)
    }

    appIdx := -1
    for i, k := range yamlNode.Content[0].Content {
        if k.Value == "applications" {
            appIdx = i + 1
            break
        }
    }

    yamlNode.Content[0].Content[appIdx].Content = append(
        yamlNode.Content[0].Content[appIdx].Content, newApp.Content[0])

    out, err := yaml.Marshal(&yamlNode)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(string(out))
}

显然,您可以从 JSON 中正确解组,而不是像我在 newApplicationNode 中那样走老路。但是,如前面的答案所述,重要的是要注意键和实际值在 Content 内的后续索引中,因此在修改文档时需要考虑到这一点。 (例如,查找 applications 键,然后考虑下一个索引(在我的示例中为 appIdx = i + 1)的内容。

希望对您有所帮助!