使用 System.Text.Json 有效替换大型 JSON 的属性

Question

我正在处理大型 JSON，其中一些元素包含以 Base 64 编码的大型（最多 100MB）文件。例如：

{ "name": "One Name", "fileContent": "...base64..." }

我想将 fileContent 属性值存储在磁盘中（以字节为单位）并将其替换为文件的路径，如下所示：

{ "name": "One Name", "fileRoute": "/route/to/file" }

是否可以 System.Text.Json 使用流或任何其他方式来避免必须在内存中处理非常大的 JSON？

Answer 1

您的基本要求是将包含属性 "fileContent": "...base64..." 的 JSON 转换为 "fileRoute": "/route/to/file"，同时将 fileContent 的值写入单独的二进制文件 而没有将 fileContent 的值具体化为完整的字符串 。

目前尚不清楚这是否可以通过 System.Text.Json 的 .NET Core 3.1 实现来完成。就算可以，也不容易。简单地从 Stream 生成 Utf8JsonReader 需要工作，参见 . Having done so, there is a method Utf8JsonReader.ValueSequence returns 最后处理的令牌的原始值作为 ReadOnlySequence<byte>输入有效载荷的切片。但是，该方法似乎并不容易使用，因为它仅在令牌包含在多个段中时才有效，不能保证值的格式正确，并且不会取消转义 JSON 转义序列。

Newtonsoft 在这里根本不起作用，因为 JsonTextReader 总是完全具体化每个原始字符串值。

作为替代方案，您可以考虑 JsonReaderWriterFactory. These readers and writers are used by DataContractJsonSerializer and translate JSON to XML on-the-fly as it is being read and written. Since the base classes for these readers and writers are XmlReader and XmlWriter, they support reading string values in chunks via XmlReader.ReadValueChunk(Char[], Int32, Int32). Even better, they support reading Base64 binary values in chunks via XmlReader.ReadContentAsBase64(Byte[], Int32, Int32) 返回的读者和作者。

给定这些读取器和写入器，我们可以使用流式转换算法将 fileContent 个节点转换为 fileRoute 个节点，同时将 Base64 二进制文件提取到单独的二进制文件中。

首先，介绍以下XML流式转换方法，粗略地基于Combining the XmlReader and XmlWriter classes for simple streaming transformations by Mark Fussell and to :

public static class XmlWriterExtensions
{
    // Adapted from this answer 
    // to 
    // By https://whosebug.com/users/3744182/dbc

    /// <summary>
    /// Make a DEEP copy of the current xmlreader node to xmlwriter, allowing the caller to transform selected elements.
    /// </summary>
    /// <param name="writer"></param>
    /// <param name="reader"></param>
    /// <param name="shouldTransform"></param>
    /// <param name="transform"></param>
    public static void WriteTransformedNode(this XmlWriter writer, XmlReader reader, Predicate<XmlReader> shouldTransform, Action<XmlReader, XmlWriter> transform)
    {
        if (reader == null || writer == null || shouldTransform == null || transform == null)
            throw new ArgumentNullException();

        int d = reader.NodeType == XmlNodeType.None ? -1 : reader.Depth;
        do
        {
            if (reader.NodeType == XmlNodeType.Element && shouldTransform(reader))
            {
                using (var subReader = reader.ReadSubtree())
                {
                    transform(subReader, writer);
                }
                // ReadSubtree() places us at the end of the current element, so we need to move to the next node.
                reader.Read();
            }
            else
            {
                writer.WriteShallowNode(reader);
            }
        }
        while (!reader.EOF && (d < reader.Depth || (d == reader.Depth && reader.NodeType == XmlNodeType.EndElement)));
    }

    /// <summary>
    /// Make a SHALLOW copy of the current xmlreader node to xmlwriter, and advance the XML reader past the current node.
    /// </summary>
    /// <param name="writer"></param>
    /// <param name="reader"></param>
    public static void WriteShallowNode(this XmlWriter writer, XmlReader reader)
    {
        // Adapted from https://docs.microsoft.com/en-us/archive/blogs/mfussell/combining-the-xmlreader-and-xmlwriter-classes-for-simple-streaming-transformations
        // By Mark Fussell https://docs.microsoft.com/en-us/archive/blogs/mfussell/
        // and rewritten to avoid using reader.Value, which fully materializes the text value of a node.
        if (reader == null)
            throw new ArgumentNullException("reader");
        if (writer == null)
            throw new ArgumentNullException("writer");

        switch (reader.NodeType)
        {   
            case XmlNodeType.None:
                // This is returned by the System.Xml.XmlReader if a Read method has not been called.
                reader.Read();
                break;

            case XmlNodeType.Element:
                writer.WriteStartElement(reader.Prefix, reader.LocalName, reader.NamespaceURI);
                writer.WriteAttributes(reader, true);
                if (reader.IsEmptyElement)
                {
                    writer.WriteEndElement();
                }
                reader.Read();
                break;

            case XmlNodeType.Text:
            case XmlNodeType.Whitespace:
            case XmlNodeType.SignificantWhitespace:
            case XmlNodeType.CDATA:
            case XmlNodeType.XmlDeclaration:
            case XmlNodeType.ProcessingInstruction:
            case XmlNodeType.EntityReference:
            case XmlNodeType.DocumentType:
            case XmlNodeType.Comment:
                //Avoid using reader.Value as this will fully materialize the string value of the node.  Use WriteNode instead,
                // it copies text values in chunks.  See: https://referencesource.microsoft.com/#system.xml/System/Xml/Core/XmlWriter.cs,368
                writer.WriteNode(reader, true);
                break;

            case XmlNodeType.EndElement:
                writer.WriteFullEndElement();
                reader.Read();
                break;

            default:
                throw new XmlException(string.Format("Unknown NodeType {0}", reader.NodeType));
        }
    }
}

public static partial class XmlReaderExtensions
{
    // Taken from this answer 
    // To 
    // By https://whosebug.com/users/3744182/dbc
    public static bool CopyBase64ElementContentsToFile(this XmlReader reader, string path)
    {
        using (var stream = File.Create(path))
        {
            byte[] buffer = new byte[8192];
            int readBytes = 0;

            while ((readBytes = reader.ReadElementContentAsBase64(buffer, 0, buffer.Length)) > 0)
            {
                stream.Write(buffer, 0, readBytes);
            }
        }
        return true;
    }
}

接下来，使用JsonReaderWriterFactory，引入以下方法从一个JSON文件流式传输到另一个文件，根据需要重写fileContent个节点：

public static class JsonPatchExtensions
{
    public static string[] PatchFileContentToFileRoute(string oldJsonFileName, string newJsonFileName, FilenameGenerator generator)
    {
        var newNames = new List<string>();

        using (var inStream = File.OpenRead(oldJsonFileName))
        using (var outStream = File.Open(newJsonFileName, FileMode.Create))
        using (var xmlReader = JsonReaderWriterFactory.CreateJsonReader(inStream, XmlDictionaryReaderQuotas.Max))
        using (var xmlWriter = JsonReaderWriterFactory.CreateJsonWriter(outStream))
        {
            xmlWriter.WriteTransformedNode(xmlReader, 
                r => r.LocalName == "fileContent" && r.NamespaceURI == "",
                (r, w) =>
                {
                    r.MoveToContent();
                    var name = generator.GenerateNewName();
                    r.CopyBase64ElementContentsToFile(name);
                    w.WriteStartElement("fileRoute", "");
                    w.WriteAttributeString("type", "string");
                    w.WriteString(name);
                    w.WriteEndElement();
                    newNames.Add(name);
                });
        }

        return newNames.ToArray();
    }
}

public abstract class FilenameGenerator
{
    public abstract string GenerateNewName();
}

// Replace the following with whatever algorithm you need to generate unique binary file names.

public class IncrementalFilenameGenerator : FilenameGenerator
{
    readonly string prefix;
    readonly string extension;
    int count = 0;

    public IncrementalFilenameGenerator(string prefix, string extension)
    {
        this.prefix = prefix;
        this.extension = extension;
    }

    public override string GenerateNewName()
    {
        var newName = Path.ChangeExtension(prefix + (++count).ToString(), extension);
        return newName;
    }
}

然后调用如下：

var binaryFileNames = JsonPatchExtensions.PatchFileContentToFileRoute(
    oldJsonFileName, 
    newJsonFileName,
    // Replace the following with your actual binary file name generation algorithm
    new IncrementalFilenameGenerator("Question59839437_fileContent_", ".bin"));

来源：

演示 fiddle here.

使用 System.Text.Json 有效替换大型 JSON 的属性

Efficiently replacing properties of a large JSON using System.Text.Json

json

.net-core

asp.net-core

system.text.json