1

I am writing some Natural Language Processing code and want to have a binary format dictionary to lookup word meanings. I originally used the standard Serialization and Binary Formatter mechanism but was horrified at how big the resulting file was, and how full of (redundant) type information. I have been trying to get the protobuf-net library to work for me but have the following problem:

When I write my dictionary to file I also put the offset that a record starts into an index file and I think that the serialization works OK, but when I look an entry up I get the correct offset but the DictionaryEntry's Definitions property contains all of the DictionaryDefinitions from then on (thousands of them). The first DictionaryDefinition in the List is the correct one for the term searched for.

I have tried implementing this as both a Property and as a Public member, to no success. I have also tried SkipConstructor = true in the ProtoContract attribute and OverwriteList = true in the ProtoMember attribute.

[ProtoContract] 
public class DictionaryEntry
{
    [ProtoMember(1)]
    public string EntryKey { get; set; }
    [ProtoMember(2)]
    public List<DictionaryDefinition> Definitions{ get; set; }
}

[ProtoContract]
public class DictionaryDefinition
{
    [ProtoMember(1)]
    public string PartOfSpeech { get; set; }
    [ProtoMember(2)]
    public string Definition { get; set; }
}

I don't think that I've done anything particularly dumb in the rest of my code, but if anyone could suggest anything for this part, I'd like to exclude this.

Thanks,

M

2
  • I should also have added: if you have existing data that you need to split, I can probably help with that - although presumably if you have the offsets you don't need that Apr 7, 2012 at 19:59
  • Actually, technically you could also calculate the length using your offset data, and deserialize the existing data using the overloads that let you supply the length manually Apr 7, 2012 at 20:00

1 Answer 1

2

The raw protobuf format has neither a length prefix not a terminator - it is designed such that append === merge.

If you need to identify individual objects inside a single stream, use the *WithLengthPrefix methods for serialize/deserialize. This will allow you to reliably pick individual elements from the stream.

1
  • Thank you for your response, that fixed it. I am amazed at your dedication and grateful for your help. I used Serializer.SerializeWithLengthPrefix(file, dictionaryEntry, PrefixStyle.Base128); and it worked like a charm.
    – mintydog
    Apr 8, 2012 at 21:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.