5

I have a Go project where I'm using Zap structured logging to log the contents of structs. That's how I initialise the logger:

zapLog, err := zap.NewProductionConfig().Build()
if err != nil {
    panic(err)
}

Initially I started with my own structs with json tags and it all worked perfectly:

zapLog.Info("Event persisted", zap.Any("event", &event))

Result:

{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
 "msg":"Event persisted","event":{"sourceType":4, "sourceId":"some-source-id", 
 "type":"updated", "value":"{...}", "context":{"foo":"bar"}}}

I now switched to protobuf and I'm struggling to achieve the same result. Initially I just got the "reflected map" version, when using zap.Any():

zapLog.Info("Event persisted", zap.Any("event", &event))
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
 "msg":"Event persisted","event":"sourceType:TYPE_X sourceId:\"some-source-id\", 
 type:\"updated\" value:{...}, context:<key: foo, value:bar>}

I tried marshalling the object with the jsonpb marshaller, which generated the correct output on itself, however, when I use it in zap.String(), the string is escaped, so I get an extra set of '\' in front of each quotation mark. Since there's processing of the logs at a later point, this causes problems there and hence I want to avoid it:

m := jsonpb.Marshaler{}
var buf bytes.Buffer
if err := m.Marshal(&buf, msg); err != nil {
    // handle error
}
zapLog.Info("Event persisted", zap.ByteString("event", buf.Bytes()))

Result:

{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
 "msg":"Event persisted","event":"{\"sourceType\":\"TYPE_X\", \"sourceId\":\"some-source-id\", 
 \"type\":\"updated\", \"value\":\"{...}\", \"context\":{\"foo\":"bar\"}}"}

I then tried using zap.Reflect() instead of zap.Any() which was the closest thing I could get to what I need, except that enums are rendered as their numerical values (the initial solution did not have enums, so that didn't work in the pre-protobuf solution either):

zapLog.Info("Event persisted", zap.Reflect("event", &event))

Result:

{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
 "msg":"Event persisted","event":{"sourceType":4, "sourceId":"some-source-id", 
 "type":"updated", "value":"{...}", "context":{"foo":"bar"}}}

The only option I see so far is to write my own MarshalLogObject() function:

type ZapEvent struct {
    event *Event
}

func (z *ZapEvent) MarshalLogObject(encoder zapcore.ObjectEncoder) error {

  encoder.AddString("sourceType", z.event.SourceType.String()
  // implement encoder for each attribute

}

func processEvent(e Event) {
   ...
   zapLog.Info("Event persisted", zap.Object("event", &ZapEvent{event: &e}))
}

But since it's a complex struct, I would rather use a less error prone and maintenance heavy solution. Ideally, I would tell zap to use the jsonpb marshaller somehow, but I don't know if that's possible.

0

2 Answers 2

4

Use zap.Any with a json.RawMessage. You can convert directly the byte output of jsonpb.Marshaler:

    foo := &pb.FooMsg{
        Foo: "blah", 
        Bar:  1,
    }

    m := jsonpb.Marshaler{}
    var buf bytes.Buffer
    if err := m.Marshal(&buf, foo); err != nil {
        // handle error
    }

    logger, _ := zap.NewDevelopment()
    logger.Info("Event persisted", zap.Any("event", json.RawMessage(buf.Bytes())))

The bytes will be printed as:

Event persisted {"event": {"foo":"blah","bar":"1"}}`

I believe that's the easiest way, however I'm also aware of a package kazegusuri/go-proto-zap-marshaler (I'm not affiliated to it) that generates MarshalLogObject() implementations as a protoc plugin. You may want to take a look at that too.

1
  • Is it possible to make it work with Desugar ? logger.Desugar().Info("Event persisted", zap.Any("event", json.RawMessage(buf.Bytes())))
    – serkan
    Commented Mar 10, 2023 at 21:22
1

I used another way to jsonify protos.

Since protos can be naturally marshaled, I just wrapped them in the strict-to-json marshaler.

And you can modify the internals to use protojson (newer jsonpb).

Unlike the marshaler in the previous solution, this one doesn't require ahead-of-logging processing.


type jsonObjectMarshaler struct {
    obj  any
}

func (j *jsonObjectMarshaler) MarshalJSON() ([]byte, error) {
    bytes, err := json.Marshal(j.obj)
    // bytes, err := protojson.Marshal(j.obj)
    if err != nil {
        return nil, fmt.Errorf("json marshaling failed: %w", err)
    }
    return bytes, nil
}

func ZapJsonable(key string, obj any) zap.Field {
    return zap.Reflect(key, &jsonObjectMarshaler{obj: obj})
}

Then to use it, just

logger, _ := zap.NewDevelopment()
logger.Info("Event persisted", ZapJsonable("event", buf))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.