84

(this is a re-post of a question that I saw in my RSS, but which was deleted by the OP. I've re-added it because I've seen this question asked several times in different places; wiki for "good form")

Suddenly, I receive a ProtoException when deserializing and the message is: unknown wire-type 6

  • What is a wire-type?
  • What are the different wire-type values and their description?
  • I suspect a field is causing the problem, how to debug this?
3
  • 11
    A Google library's Exception message sent me here. That's pretty funny...
    – Zach Smith
    Commented Nov 27, 2017 at 16:37
  • 2
    @Zach was it actually the google library? I know I link from protobuf-net, but that isn't google. It's an independent implementation of a google protocol. Commented Nov 27, 2017 at 19:13
  • Easy thing to check before you look any further: Are both ends using matching classes? Mine were not, in my case because because I had accidentally made the wrong call to the server!
    – Timo
    Commented Aug 7, 2018 at 13:34

10 Answers 10

69

First thing to check:

IS THE INPUT DATA PROTOBUF DATA? If you try and parse another format (json, xml, csv, binary-formatter), or simply broken data (an "internal server error" html placeholder text page, for example), then it won't work.


What is a wire-type?

It is a 3-bit flag that tells it (in broad terms; it is only 3 bits after all) what the next data looks like.

Each field in protocol buffers is prefixed by a header that tells it which field (number) it represents, and what type of data is coming next; this "what type of data" is essential to support the case where unanticipated data is in the stream (for example, you've added fields to the data-type at one end), as it lets the serializer know how to read past that data (or store it for round-trip if required).

What are the different wire-type values and their description?

  • 0: variant-length integer (up to 64 bits) - base-128 encoded with the MSB indicating continuation (used as the default for integer types, including enums)
  • 1: 64-bit - 8 bytes of data (used for double, or electively for long/ulong)
  • 2: length-prefixed - first read an integer using variant-length encoding; this tells you how many bytes of data follow (used for strings, byte[], "packed" arrays, and as the default for child objects properties / lists)
  • 3: "start group" - an alternative mechanism for encoding child objects that uses start/end tags - largely deprecated by Google, it is more expensive to skip an entire child-object field since you can't just "seek" past an unexpected object
  • 4: "end group" - twinned with 3
  • 5: 32-bit - 4 bytes of data (used for float, or electively for int/uint and other small integer types)

I suspect a field is causing the problem, how to debug this?

Are you serializing to a file? The most likely cause (in my experience) is that you have overwritten an existing file, but have not truncated it; i.e. it was 200 bytes; you've re-written it, but with only 182 bytes. There are now 18 bytes of garbage on the end of your stream that is tripping it up. Files must be truncated when re-writing protocol buffers. You can do this with FileMode:

using(var file = new FileStream(path, FileMode.Truncate)) {
    // write
}

or alternatively by SetLength after writing your data:

file.SetLength(file.Position);

Other possible cause

You are (accidentally) deserializing a stream into a different type than what was serialized. It's worth double-checking both sides of the conversation to ensure this is not happening.

16
  • See the docs here: code.google.com/apis/protocolbuffers/docs/… Commented Feb 16, 2012 at 19:00
  • @Nigel I'm very familiar with the spec; the problem described here, though, relates to accidental corruption by not trimming a file. Commented Feb 16, 2012 at 19:14
  • 21
    +1 Nigel, leading protobuf-net's creator to the protobuf specs :), like a boss! I got the exception but it wasn't file related. It was because I had updated one protobuffed object on the client and forgot to publish to the web server. (Not sure if this is worth adding to possible causes)
    – Joe
    Commented Apr 26, 2012 at 3:44
  • 2
    @MarcGravell just a follow up on your comments. I received exactly this exception when sending side was using a class class Foo { DateTime OccurredOn { get; set; } } and receiving side had class Foo { long OccurredOn { get; set; } }. Notice the difference between object's types: DateTime vs long. Not sure if you want to address this, but this supports @Joe findings. I use protobuf-net 2.0.0.668. I fixed it by using same type on both ends (which make sense), but the exception in this case was not related to a file.
    – oleksii
    Commented Jun 25, 2014 at 13:19
  • 1
    @oleksii yeah, that would indeed be an invalid wire type; fair enough. Commented Jun 25, 2014 at 13:23
49

Since the stack trace references this StackOverflow question, I thought I'd point out that you can also receive this exception if you (accidentally) deserialize a stream into a different type than what was serialized. So it's worth double-checking both sides of the conversation to ensure this is not happening.

3
  • 10
    Fair point - and probably - no, definitely - the first thing to check. Commented Jun 15, 2013 at 22:39
  • 3
    Yeah, that's what I just accidentally did -- changed the type of an existing field. Well, this here is quite an obvious mistake, but still it took me some minutes to realize what I did wrong (required inspecting some diffs of proto files and stuff). Is there a way to have a full log of deserialization? It could quickly give me an idea of where to look for the broken field. Commented Dec 23, 2013 at 9:37
  • In my case the different type just the same type but not nullable
    – Ahmad
    Commented Apr 28, 2022 at 8:44
11

This can also be caused by an attempt to write more than one protobuf message to a single stream. The solution is to use SerializeWithLengthPrefix and DeserializeWithLengthPrefix.


Why this happens:

The protobuf specification supports a fairly small number of wire-types (the binary storage formats) and data-types (the .NET etc data-types). Additionally, this is not 1:1, nor is is 1:many or many:1 - a single wire-type can be used for multiple data-types, and a single data-type can be encoded via any of multiple wire-types. As a consequence, you cannot fully understand a protobuf fragment unless you already know the scema, so you know how to interpret each value. When you are, say, reading an Int32 data-type, the supported wire-types might be "varint", "fixed32" and "fixed64", where-as when reading a String data-type, the only supported wire-type is "string".

If there is no compatible map between the data-type and wire-type, then the data cannot be read, and this error is raised.

Now let's look at why this occurs in the scenario here:

[ProtoContract]
public class Data1
{
    [ProtoMember(1, IsRequired=true)]
    public int A { get; set; }
}

[ProtoContract]
public class Data2
{
    [ProtoMember(1, IsRequired = true)]
    public string B { get; set; }
}

class Program
{
    static void Main(string[] args)
    {
        var d1 = new Data1 { A = 1};
        var d2 = new Data2 { B = "Hello" };
        var ms = new MemoryStream();
        Serializer.Serialize(ms, d1); 
        Serializer.Serialize(ms, d2);
        ms.Position = 0;
        var d3 = Serializer.Deserialize<Data1>(ms); // This will fail
        var d4 = Serializer.Deserialize<Data2>(ms);
        Console.WriteLine("{0} {1}", d3, d4);
    }
}

In the above, two messages are written directly after each-other. The complication is: protobuf is an appendable format, with append meaning "merge". A protobuf message does not know its own length, so the default way of reading a message is: read until EOF. However, here we have appended two different types. If we read this back, it does not know when we have finished reading the first message, so it keeps reading. When it gets to data from the second message, we find ourselves reading a "string" wire-type, but we are still trying to populate a Data1 instance, for which member 1 is an Int32. There is no map between "string" and Int32, so it explodes.

The *WithLengthPrefix methods allow the serializer to know where each message finishes; so, if we serialize a Data1 and Data2 using the *WithLengthPrefix, then deserialize a Data1 and a Data2 using the *WithLengthPrefix methods, then it correctly splits the incoming data between the two instances, only reading the right value into the right object.

Additionally, when storing heterogeneous data like this, you might want to additionally assign (via *WithLengthPrefix) a different field-number to each class; this provides greater visibility of which type is being deserialized. There is also a method in Serializer.NonGeneric which can then be used to deserialize the data without needing to know in advance what we are deserializing:

// Data1 is "1", Data2 is "2"
Serializer.SerializeWithLengthPrefix(ms, d1, PrefixStyle.Base128, 1);
Serializer.SerializeWithLengthPrefix(ms, d2, PrefixStyle.Base128, 2);
ms.Position = 0;

var lookup = new Dictionary<int,Type> { {1, typeof(Data1)}, {2,typeof(Data2)}};
object obj;
while (Serializer.NonGeneric.TryDeserializeWithLengthPrefix(ms,
    PrefixStyle.Base128, fieldNum => lookup[fieldNum], out obj))
{
    Console.WriteLine(obj); // writes Data1 on the first iteration,
                            // and Data2 on the second iteration
}
9
  • 2
    Hmmmm.... no, that shouldn't cause this error. The protobuf spec is designed to be appendable, so writing a second object would result in a perfectly valid and legal protobuf stream, so should not cause this error (which refers to a corrupt stream). The significance of *WithLengthPrefix is that without it, you have absolutely no way of knowing where the first message ends and the second message begins. If you have an example that shows multiple messages being written causing this error, I would very much like to see it. Commented Sep 13, 2012 at 11:20
  • I was serializing two different objects in sequence to the same stream using two calls to Serialize (it was a key-value pair). Deserializing causes the unknown wire-type error. I had reused the same ids in both objects (each starts from 1). Putting two of the same obect in the same stream leaves the stream at the end after the first deserialize (returning you the data of the second object) the second read gives you a default object. Commented Sep 13, 2012 at 12:01
  • I'll need to repro it, but I strongly suspect that is a different-but-similar message, in particular that the discovered wire-type is not valid for the expected data. Which would make since, because you are essentially merging a string and an int. I'll check later, though. Commented Sep 16, 2012 at 17:18
  • 1
    I apologise; it does indeed raise the same message, but the context is different. If you don't mind, I'll add some more context to your answer. Commented Sep 17, 2012 at 9:21
  • 1
    @Philip try on Serializer.NonGeneric Commented Feb 19, 2021 at 11:32
5

Previous answers already explain the problem better than I can. I just want to add an even simpler way to reproduce the exception.

This error will also occur simply if the type of a serialized ProtoMember is different from the expected type during deserialization.

For instance if the client sends the following message:

public class DummyRequest
{
    [ProtoMember(1)]
    public int Foo{ get; set; }
}

But what the server deserializes the message into is the following class:

public class DummyRequest
{
    [ProtoMember(1)]
    public string Foo{ get; set; }
}

Then this will result in the for this case slightly misleading error message

ProtoBuf.ProtoException: Invalid wire-type; this usually means you have over-written a file without truncating or setting the length

It will even occur if the property name changed. Let's say the client sent the following instead:

public class DummyRequest
{
    [ProtoMember(1)]
    public int Bar{ get; set; }
}

This will still cause the server to deserialize the int Bar to string Foo which causes the same ProtoBuf.ProtoException.

I hope this helps somebody debugging their application.

0
1

Also check the obvious that all your subclasses have [ProtoContract] attribute. Sometimes you can miss it when you have rich DTO.

2
  • 1
    That would usually present a different error message, though (about an unexpected type) Commented Jun 16, 2014 at 11:01
  • @MarcGravell Once I have SerializationException: ProtoBufServiceClient: Error deserializing: Invalid wire-type; and this page address but sometime, when I don't put attribute I have: SerializationException: ProtoBufServiceClient: Error deserializing: No serializer defined for type: Foo ---> System.InvalidOperationException: No serializer defined for type: Foo. In first case I have: class A { public List<Bar> bars {get;set;}}, in second it's single field of type Foo. I'll inspect the problem in simple console app.
    – Tomasito
    Commented Jun 16, 2014 at 12:23
1

I've seen this issue when using the improper Encoding type to convert the bytes in and out of strings.

Need to use Encoding.Default and not Encoding.UTF8.

using (var ms = new MemoryStream())
{
    Serializer.Serialize(ms, obj);
    var bytes = ms.ToArray();
    str = Encoding.Default.GetString(bytes);
}
4
  • 3
    Need to use Encoding.Default ... no, just no; you never - never - want to use Encoding.Default for anything. Encoding is simply incorrect here; if you need binary as text, use base-64 or similar; not an encoding. And encoding is for storing text as binary, not the other way around. Commented Aug 6, 2018 at 16:00
  • i would like all people reading this to know that @MarcGravell is orders of magnitude better than i, please accept his answer, thanks for not downvoting mine. even though its worked for us in production for years in our code that serializes objects out to redis. hopefully that doesnt make mark double shudder. he deserves better. he's earned it. thanks, community
    – Micah
    Commented Aug 7, 2018 at 18:09
  • did you know redis can talk binary? You can just give redis your byte[] Commented Aug 7, 2018 at 19:56
  • I had same issue with using Encoding.UTF8. Solved it by converting from and to Base64String.
    – Jins Peter
    Commented Sep 21, 2020 at 17:51
1

If you are using SerializeWithLengthPrefix, please mind that casting instance to object type breaks the deserialization code and causes ProtoBuf.ProtoException : Invalid wire-type.

using (var ms = new MemoryStream())
{
    var msg = new Message();
    Serializer.SerializeWithLengthPrefix(ms, (object)msg, PrefixStyle.Base128); // Casting msg to object breaks the deserialization code.
    ms.Position = 0;
    Serializer.DeserializeWithLengthPrefix<Message>(ms, PrefixStyle.Base128)
}
1
  • well, I guess that's true... but... why would you do that? btw, there's a non-generic API in Serializer.NonGeneric that will work fine with object Commented Sep 29, 2017 at 13:52
1

This happened in my case because I had something like this:

var ms = new MemoryStream();
Serializer.Serialize(ms, batch);

_queue.Add(Convert.ToBase64String(ms.ToArray()));

So basically I was putting a base64 into a queue and then, on the consumer side I had:

var stream = new MemoryStream(Encoding.UTF8.GetBytes(myQueueItem));
var batch = Serializer.Deserialize<List<EventData>>(stream);

So though the type of each myQueueItem was correct, I forgot that I converted a string. The solution was to convert it once more:

var bytes = Convert.FromBase64String(myQueueItem);
var stream = new MemoryStream(bytes);
var batch = Serializer.Deserialize<List<EventData>>(stream);
1

It can also happen when you serialize ProtoContract vs. 1 and then deserialize the result with another ProtoContract vs. 2.

E.g. you have changed aProtoMember's number of order or perhaps it's type.
You can however add ProtoMember's with new numbers without breaking the deserialiazation.

1
  • well, I mean technically true, but also sort of predicable... this is using an incorrect model, so indeed: it may encounter unexpected things Commented Jan 22 at 14:32
0
using (var fs = File.OpenRead(source))
using (var brotliStream = new BrotliStream(fs, CompressionMode.Fast))
 {
    var index = Serializer.Deserialize<List<Files>>(fs);  <--- Error
 }

In my case the Brotli stream was using source stream. Exception got fixed once I placed Brotlistream after Deserialization

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.