protobuf-net 集合序列化抛出 StackOverflowException

Question

我有一个使用 protobuf-net 序列化的嵌入式 C# 应用程序。当序列化大约 50 个条目的集合时，它会抛出 WhosebugException，只能在运行 WinCE 的设备上重现。堆栈大约有 600 个条目，结尾如下：

at ProtoBuf.Meta.RuntimeTypeModel.GetKey(Type type, Boolean demand, Boolean getBaseKey)
   at ProtoBuf.Meta.ValueMember.TryGetCoreSerializer(RuntimeTypeModel model, DataFormat dataFormat, Type type, WireType& defaultWireType, Boolean asReference, Boolean dynamicType, Boolean overwriteList, Boolean allowComplexTypes)
   at ProtoBuf.Meta.ValueMember.BuildSerializer()
   at ProtoBuf.Meta.ValueMember.get_Serializer()
   at ProtoBuf.Meta.MetaType.BuildSerializer()
   at ProtoBuf.Meta.MetaType.get_Serializer()
   at ProtoBuf.Meta.RuntimeTypeModel.Serialize(Int32 key, Object value, ProtoWriter dest)
   at ProtoBuf.ProtoWriter.WriteObject(Object value, Int32 key, ProtoWriter writer)
   at ProtoBuf.Serializers.SubItemSerializer.ProtoBuf.Serializers.IProtoSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TagDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.PropertyDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TypeSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Meta.RuntimeTypeModel.Serialize(Int32 key, Object value, ProtoWriter dest)
   at ProtoBuf.ProtoWriter.WriteObject(Object value, Int32 key, ProtoWriter writer)
   at ProtoBuf.BclHelpers.WriteNetObject(Object value, ProtoWriter dest, Int32 key, NetObjectOptions options)
   at ProtoBuf.Serializers.NetObjectSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TagDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.FieldDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TypeSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Meta.RuntimeTypeModel.Serialize(Int32 key, Object value, ProtoWriter dest)
   at ProtoBuf.ProtoWriter.WriteObject(Object value, Int32 key, ProtoWriter writer)
   at ProtoBuf.Serializers.SubItemSerializer.ProtoBuf.Serializers.IProtoSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TagDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.ListDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.PropertyDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TypeSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Meta.RuntimeTypeModel.Serialize(Int32 key, Object value, ProtoWriter dest)
   at ProtoBuf.ProtoWriter.WriteObject(Object value, Int32 key, ProtoWriter writer)
   at ProtoBuf.BclHelpers.WriteNetObject(Object value, ProtoWriter dest, Int32 key, NetObjectOptions options)
   at ProtoBuf.Serializers.NetObjectSerializer.Write(Object value, ProtoWriter dest)

列表定义如下：

[ProtoMember(2, AsReference = true, DataFormat = DataFormat.Group)]
        //[ProtoMember(2, AsReference = true)]
        public NodeList<T> Nodes { get; private set; }

（我尝试了两个版本，有和没有DataFormat.Group）

[ProtoContract(ImplicitFields = ImplicitFields.AllFields)]        
public class NodeList<T> : Collection<GraphNode<T>> where T : new()

当运行应用在Windows 7资源较多时不抛出异常

有没有优化序列化的可能性？我做错了什么吗？

为什么调用栈这么长？我认为列表应该被视为顺序数组，而不是递归调用？

谢谢，黛安娜

Answer 1

您没有在问题中包含 GraphNode<T> 的示例。但是，从名字上看，它似乎是一个通用的图节点元素，类似于微软技术文章An Extensive Examination of Data Structures Using C# 2.0: Part 5: From Trees to Graphs.

中的GraphNode<T>、NodeList<T>和Graph<T>

在那种情况下，如果你的节点图很深，根节点在集合的开始，那么很有可能超过最大堆栈深度序列化节点列表。原因在 Protobuf-net: the unofficial manual:

中有解释

How references work

When serializing a class Foo to which AsReference or AsReferenceDefault applies, the type of the field in the protocol buffer changes from Foo to bcl.NetObjectProxy, which is defined as follows in the source code of protobuf-net (Tools/bcl.proto):
message NetObjectProxy {
  // for a tracked object, the key of the **first** 
  // time this object was seen
  optional int32 existingObjectKey = 1; 
  // for a tracked object, a **new** key, the first 
  // time this object is seen
  optional int32 newObjectKey = 2;
  // for dynamic typing, the key of the **first** time 
  // this type was seen
  optional int32 existingTypeKey = 3; 
  // for dynamic typing, a **new** key, the first time 
  // this type is seen
  optional int32 newTypeKey = 4;
  // for dynamic typing, the name of the type (only 
  // present along with newTypeKey) 
  optional string typeName = 8;
  // the new string/value (only present along with 
  // newObjectKey)
  optional bytes payload = 10;
}
So it appears that

The first time an object is encountered, the newObjectKey and a payload fields are written; presumably, the payload is stored as if its type is Foo.

When the object is encountered again, just the existingObjectKey is written.

因此，如果图恰好很深并且遇到根作为列表中的第一个元素，protobuf-net 将递归地深度优先遍历图而不是广度优先遍历数组，并且从而溢出堆栈。

为避免此问题，您可以利用以下事实：当 AsReference = true 时，protobuf-net 仅在第一次遇到对象时遍历对象的成员，以序列化您的 NodeList<T>分两个阶段：

在没有邻居信息的情况下序列化节点列表。
然后序列化一个table的邻居信息。

例如，使用上述文章中GraphNode<T>、NodeList<T>和Graph<T>的简化版本作为基础，可以按如下方式完成：

[ProtoContract]
public class GraphNode<T> where T : new()
{
    readonly NodeList<T> neighbors = new NodeList<T>();

    public GraphNode()
    {
        this.Value = new T();
    }

    public GraphNode(T value)
        : this()
    {
        this.Value = value;
    }

    [ProtoMember(1, AsReference = true)]
    public T Value { get; set; }

    // Do not serialize the list of neighbors directly!  
    // Instead this will be serialized by the NodeList<T> owned by the Graph<T>
    [ProtoIgnore]
    public NodeList<T> Neighbors { get { return neighbors; } }
}

[ProtoContract(IgnoreListHandling = true)]
public class NodeList<T> : Collection<GraphNode<T>> where T : new()
{
    [ProtoContract]
    class GraphNodeNeighborsProxy
    {
        [ProtoMember(1, AsReference = true)]
        public GraphNode<T> Node { get; set; }

        [ProtoMember(2, AsReference = true, DataFormat = DataFormat.Group)]
        public ICollection<GraphNode<T>> Neighbors
        {
            get
            {
                return Node == null ? null : Node.Neighbors;
            }
        }
    }

    [ProtoMember(1, AsReference = true, DataFormat = DataFormat.Group)]
    IEnumerable<GraphNode<T>> Nodes
    {
        get
        {
            return new SerializationCollectionWrapper<GraphNode<T>, GraphNode<T>>(this, n => n, (c, n) => c.Add(n)); 
        }
    }

    [ProtoMember(2, DataFormat = DataFormat.Group)]
    IEnumerable<GraphNodeNeighborsProxy> NeighborsTable
    {
        get
        {
            return new SerializationCollectionWrapper<GraphNode<T>, GraphNodeNeighborsProxy>(
                this,
                n => new GraphNodeNeighborsProxy { Node = n },
                (c, proxy) => {}
                );
        }
    }
}

[ProtoContract]
public class Graph<T> where T : new()
{
    readonly private NodeList<T> nodeSet = new NodeList<T>();

    public Graph() { }

    public GraphNode<T> AddNode(GraphNode<T> node)
    {
        // adds a node to the graph
        nodeSet.Add(node);
        return node;
    }

    public GraphNode<T> AddNode(T value)
    {
        // adds a node to the graph
        return AddNode(new GraphNode<T>(value));
    }

    public void AddDirectedEdge(GraphNode<T> from, GraphNode<T> to)
    {
        from.Neighbors.Add(to);
    }

    [ProtoMember(2, AsReference = true, DataFormat = DataFormat.Group)]
    public NodeList<T> Nodes
    {
        get
        {
            return nodeSet;
        }
    }
}

public class SerializationCollectionWrapper<TFrom, TTo> : ICollection<TTo>
{
    readonly ICollection<TFrom> collection;
    readonly Func<TFrom, TTo> mapTo;
    readonly Action<ICollection<TFrom>, TTo> add;

    public SerializationCollectionWrapper(ICollection<TFrom> collection, Func<TFrom, TTo> mapTo, Action<ICollection<TFrom>, TTo> add)
    {
        if (collection == null || mapTo == null || add == null)
            throw new ArgumentNullException();
        this.collection = collection;
        this.mapTo = mapTo;
        this.add = add;
    }

    ICollection<TFrom> Collection { get { return collection; } }

    #region ICollection<TTo> Members

    public void CopyTo(TTo[] array, int arrayIndex)
    {
        foreach (var item in this)
            array[arrayIndex++] = item;
    }

    public int Count
    {
        get { return Collection.Count; }
    }

    public bool IsReadOnly
    {
        get { return Collection.IsReadOnly; }
    }

    public void Add(TTo item)
    {
        add(Collection, item);
    }

    public void Clear()
    {
        throw new NotImplementedException();
    }

    public bool Contains(TTo item)
    {
        throw new NotImplementedException();
    }

    public bool Remove(TTo item)
    {
        throw new NotImplementedException();
    }

    #endregion

    #region IEnumerable<TTo> Members

    public IEnumerator<TTo> GetEnumerator()
    {
        foreach (var item in Collection)
            yield return mapTo(item);
    }

    #endregion

    #region IEnumerable Members

    System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
    {
        return GetEnumerator();
    }

    #endregion
}

请注意，这仅在根节点列表包含图中的所有节点时才有效。如果您没有图表中所有节点的 table，则需要计算图表的 transitive closure 以预先序列化所有节点。

protobuf-net 集合序列化抛出 StackOverflowException

protobuf-net collection serialization throws StackOverflowException

c#

serialization

protobuf-net

windows-ce