368

I'm recently using gRPC with proto3, and I've noticed that required and optional has been removed in new syntax.

Would anyone kindly explain why required/optional are removed in proto3? Such kind of constraints just seem necessary to make definition robust.

syntax proto2:

message SearchRequest {
  required string query = 1;
  optional int32 page_number = 2;
  optional int32 result_per_page = 3;
}

syntax proto3:

syntax = "proto3";
message SearchRequest {
  string query = 1;
  int32 page_number = 2;
  int32 result_per_page = 3;
}

4 Answers 4

623

The usefulness of required has been at the heart of many a debate and flame war. Large camps have existed on both sides. One camp liked guaranteeing a value was present and was willing to live with its limitations but the other camp felt required dangerous or unhelpful as it can't be safely added nor removed.

Let me explain more of the reasoning why required fields should be used sparingly. If you are already using a proto, you can't add a required field because old application's won't be providing that field and applications in general don't handle the failure well. You can make sure that all old applications are upgraded first, but it can be easy to make a mistake and it doesn't help if you are storing the protos in any datastore (even short-lived, like memcached). The same sort of situation applies when removing a required field.

Many required fields were "obviously" required until... they weren't. Let's say you have an id field for a Get method. That is obviously required. Except, later you might need to change the id from int to string, or int32 to int64. That requires adding a new muchBetterId field, and now you are left with the old id field that must be specified, but eventually is completely ignored.

When those two problems are combined, the number of beneficial required fields becomes limited and the camps argue over whether it still has value. The opponents of required weren't necessarily against the idea, but its current form. Some suggested developing a more expressive validation library that could check required along with something more advanced like name.length > 10, while also making sure to have a better failure model.

Proto3 overall seems to favor simplicity, and required removal is simpler. But maybe more convincing, removing required made sense for proto3 when combined with other features, like removal of field presence for primitives and removal of overriding default values.

I'm not a protobuf developer and am in no way authoritative on the subject, but I still hope that the explanation is useful.

17
  • 39
    Yep. See also this extended explanation of things that can go horribly wrong with required fields: capnproto.org/… Aug 6, 2015 at 2:25
  • 17
    Optional isn't removed; everything is optional in proto3. But yes, field visibility (has_field) has been removed for primitives. If you need field visibility use wrappers.proto which has messages like StringValue. Since they are messages, has_field is available. This is effectively "boxing" which is common in many languages. Jan 11, 2017 at 0:31
  • 20
    On the contrary, it seems like "optional" was removed in proto3. Every field exists, and is filled in with a default value. You have no way of knowing if the primitive field was filled in by the user, or by default. Message fields, which are basically pointers, are optional, in that they can have a null value.
    – Vagrant
    Jan 24, 2017 at 1:12
  • 37
    i feel like protobuf is a language designed expressly to start flame wars
    – Randy L
    Feb 6, 2018 at 19:53
  • 18
    Seems like most people don't want to version their API's. It's easier for them to make everything optional for "backward compatibility".
    – Holoceo
    Nov 9, 2018 at 8:05
88

You can find the explanation in this protobuf Github issue:

We dropped required fields in proto3 because required fields are generally considered harmful and violating protobuf's compatibility semantics. The whole idea of using protobuf is that it allows you to add/remove fields from your protocol definition while still being fully forward/backward compatible with newer/older binaries. Required fields break this though. You can never safely add a required field to a .proto definition, nor can you safely remove an existing required field because both of these actions break wire compatibility. For example, if you add a required field to a .proto definition, binaries built with the new definition won't be able to parse data serialized using the old definition because the required field is not present in old data. In a complex system where .proto definitions are shared widely across many different components of the system, adding/removing required fields could easily bring down multiple parts of the system. We have seen production issues caused by this multiple times and it's pretty much banned everywhere inside Google for anyone to add/remove required fields. For this reason we completely removed required fields in proto3.

After the removal of "required", "optional" is just redundant so we removed "optional" as well.

13
  • 18
    I don't get it; what's the difference between dropping a message after deserializing and on deserialization? it will get dropped by the older client since it does not contains a field that is needed (e.g. id).
    – Shmuel H.
    Jan 3, 2019 at 7:07
  • 14
    I'm inclined to agree with @ShmuelH. required fields are going to be a part of an api one way or another. Well that's supported automatically through syntax given to both parties, or hidden in the backend, it's still there. May as well make it visible in the api definition
    – Cruncher
    Jul 3, 2019 at 15:27
  • 19
    I totally agree with @ShmuelH. fields are required in an API one way or another and its useful for the customer to know this. This makes me think we just haven't gotten versioning right yet. Sep 6, 2019 at 0:41
  • 15
    Another vote for @ShmuelH. If you change your API in a backwards-incompatible way (adding a required field), then surely you want your parser to detect that? Version your APIs! You can even do it completely in Protobuf if you want, using oneof { MessageV1, MessageV2, etc. }.
    – Timmmm
    Dec 4, 2019 at 11:05
  • 4
    It really boils down to the fact that required/optional forces the bundling of two concerns; the first is serialization and the second is application data validation. This bundling leads to the problems described, meaning that you really want validation to happen somewhere else. It is an error to have the serialization layer include this validation, though ideally you want it to be easily interoperable. Mar 4, 2021 at 1:01
35

Optional fields were returned in protobuf 3.15

3
  • 7
    If everything is optional, then what is the use of returning "optional" in the said version?
    – C--
    Mar 30, 2021 at 14:23
  • 3
    @SubinSebastian see github.com/protocolbuffers/protobuf/blob/master/docs/… Apr 9, 2021 at 13:10
  • 20
    @SubinSebastian with optional you get the ability to explicitly check if a field is set. Lets say you have a field int32 confidence. Currently when receiving a message which such a type you cannot know the difference between confidence = 0 or confidence not set. Because default values are optimized away in the serialization. If you mark the field as optional then presumably some extra bits are set in the serialization and a has_confidence() method will be generated so that you on the receiving end can disambiguate the two.
    – Alex Telon
    May 20, 2021 at 7:57
5

Because orthogonal factoring of related concepts is hard and the Protocol Buffers design combined at least 4 separate concepts in a way that became frustrating: Nullability (a.k.a. Presence Tracking), Content Validation, Useful Defaults, and Space Efficiency.

Proto2 allowed a field to be either 'required' or 'optional' and allowed specifying default values, but only for 'optional' fields.

The 'optional' keyword came back in 3.12/3.15 to be used for Presence Tracking. See the app note on Field Presence.

The 'required' keyword had been applied as a validation check and that was just unfortunate because Protobuf isn't up to the task of being a validation tool. It doesn't have min/max value syntax, length, or pattern restrictions. More importantly it has nothing for expressing data value dependencies between fields (apart from relationships intrinsic to basic structure).

Protobuf uses 'default' values for initial construction values and as a mechanism to reduce the size of messages by not sending those values(in proto3). That is a bit unfortunate because being able to specify an initial value for a particular code generation run would be handy for either a producer ("this is the value I want to send in my normal usage") or a consumer ("maybe I should check for null/not present, but falling back as if 'x' was sent will be good enough for now").

OTOH, it does seem like it would be helpful to specify a contractual default assumption for how to proceed if a value is not supplied. However, doing a good job of that often depends on the values(or presence) of other fields and Protobuf isn't up to that for the same (arguably beautiful) lack of complexity that makes it unsuitable for data validation. So if you want good behavior in the face of missing fields, you are best off combining explicit presence checks with whatever other data checks are appropriate. .... and at least in 3.12/3.15 onward you can. ... and maybe 0-ish values are good enough for the simplistic cases.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.