fastavro

Question

对 avro 和 python 有点陌生。

我正在尝试使用 fastavro 库简单地转换为 avro，因为本机 apache avro 库的速度有点太慢了。

我想：

1.Take 一个 json 文件 2.将数据转换成avro。

我的问题是我的 json 格式似乎不正确，无法转换为 avro。我什至尝试将我的 json 放入一个字符串变量中，并使其看起来类似于他们在网站上的语法 @ https://fastavro.readthedocs.io/en/latest/writer.html:

    {u'station': u'011990-99999', u'temp': 22, u'time': 1433270389},
    {u'station': u'011990-99999', u'temp': -11, u'time': 1433273379},
    {u'station': u'012650-99999', u'temp': 111, u'time': 1433275478},

这是我的代码：

from fastavro import json_writer, parse_schema, writer
import json

key = "test.json"
schemaFileName = "test_schema.avsc"
with open(r'C:/Path/to/file' + schemaFileName) as sc:
    w = json.load(sc)

schema = parse_schema(w)

with open(r'C:/Path/to/file/' + key) as js:
    x=json.load(js)

with open('C:/Path/to/file/output.avro', 'wb') as out:
    writer(out, schema,x, codec='deflate')

这是我得到的输出：

  File "avropython.py", line 26, in <module>
    writer(out, schema,x, codec='deflate')
  File "fastavro\_write.pyx", line 608, in fastavro._write.writer
ValueError: "records" argument should be an iterable, not dict

我的 json 文件和架构，分别是：

  "joined": false,
  "toward": {
    "selection": "dress",
    "near": true,
    "shoulder": false,
    "fine": -109780201.3804388,
    "pet": {
      "stood": "saddle",
      "live": false,
      "leather": false,
      "tube": false,
      "over": false,
      "impossible": true
    },
    "higher": false
  },
  "wear": true,
  "asleep": "door",
  "connected": true,
  "stairs": -1195512399.5000324
}

{
  "name": "MyClass",
  "type": "record",
  "namespace": "com.acme.avro",
  "fields": [
    {
      "name": "joined",
      "type": "boolean"
    },
    {
      "name": "toward",
      "type": {
        "name": "toward",
        "type": "record",
        "fields": [
          {
            "name": "selection",
            "type": "string"
          },
          {
            "name": "near",
            "type": "boolean"
          },
          {
            "name": "shoulder",
            "type": "boolean"
          },
          {
            "name": "fine",
            "type": "float"
          },
          {
            "name": "pet",
            "type": {
              "name": "pet",
              "type": "record",
              "fields": [
                {
                  "name": "stood",
                  "type": "string"
                },
                {
                  "name": "live",
                  "type": "boolean"
                },
                {
                  "name": "leather",
                  "type": "boolean"
                },
                {
                  "name": "tube",
                  "type": "boolean"
                },
                {
                  "name": "over",
                  "type": "boolean"
                },
                {
                  "name": "impossible",
                  "type": "boolean"
                }
              ]
            }
          },
          {
            "name": "higher",
            "type": "boolean"
          }
        ]
      }
    },
    {
      "name": "wear",
      "type": "boolean"
    },
    {
      "name": "asleep",
      "type": "string"
    },
    {
      "name": "connected",
      "type": "boolean"
    },
    {
      "name": "stairs",
      "type": "float"
    }
  ]
}

如果有人能帮助我，将不胜感激！！

Answer 1

如错误 ValueError: "records" argument should be an iterable, not dict 中所述，问题是当您调用 writer 时，记录的参数需要是可迭代的。解决此问题的一种方法是将最后一行更改为 writer(out, schema, [x], codec='deflate')

或者，有一个 schemaless_writer 可以用来只写一条记录：https://fastavro.readthedocs.io/en/latest/writer.html#fastavro._write_py.schemaless_writer

fastavro - 将 json 文件转换为 avro 文件

fastavro - Convert json file into avro file

python

avro