282

I've created an API in Go that, upon being called, performs a query, creates an instance of a struct, and then encodes that struct as JSON before sending back to the caller. I'd now like to allow the caller to be able to select the specific fields they would like returned by passing in a "fields" GET parameter.

This means depending on the fields value(s), my struct would change. Is there any way to remove fields from a struct? Or at least hide them in the JSON response dynamically? (Note: Sometimes I have empty values so the JSON omitEmpty tag will not work here) If neither of these are possible, is there a suggestion on a better way to handle this?

A smaller version of the structs I'm using are below:

type SearchResult struct {
    Date        string      `json:"date"`
    IdCompany   int         `json:"idCompany"`
    Company     string      `json:"company"`
    IdIndustry  interface{} `json:"idIndustry"`
    Industry    string      `json:"industry"`
    IdContinent interface{} `json:"idContinent"`
    Continent   string      `json:"continent"`
    IdCountry   interface{} `json:"idCountry"`
    Country     string      `json:"country"`
    IdState     interface{} `json:"idState"`
    State       string      `json:"state"`
    IdCity      interface{} `json:"idCity"`
    City        string      `json:"city"`
} //SearchResult

type SearchResults struct {
    NumberResults int            `json:"numberResults"`
    Results       []SearchResult `json:"results"`
} //type SearchResults

I then encode and output the response like so:

err := json.NewEncoder(c.ResponseWriter).Encode(&msg)
1
  • 7
    @Jacob, as per PuerkitoBio's updated answer, I think you mis-read the question. The (currently) accepted might not be the "correct answer" to your question, but is to the one asked here! The (currently) highest voted answer may answer your question but is completely inapplicable to this one!
    – Dave C
    Mar 29, 2015 at 19:06

15 Answers 15

459

The question is asking for fields to be dynamically selected based on the caller-provided list of fields. This isn't possible to be done with the statically-defined json struct tag.

If what you want is to always skip a field to json-encode, then of course use json:"-" to ignore the field. (Note also that this is not required if your field is unexported; those fields are always ignored by the json encoder.) This isn't what the question asks.

To quote the comment on the json:"-" answer:

This [the json:"-" answer] is the answer most people ending up here from searching would want, but it's not the answer to the question.

I'd use a map[string]interface{} instead of a struct in this case. You can easily remove fields by calling the delete built-in on the map for the fields to remove.

That is, if you can't query only for the requested fields in the first place.

7
  • 9
    you most likely don't want to throw away your type definition entirely. That's going to be bothersome down the line, like when you want to write other methods on this type that access those fields. Using an intermediate map[string]interface{} does make sense, but it does not require that you throw away your type definition.
    – jorelli
    Jun 26, 2013 at 14:47
  • 2
    The other answer is the actual answer to this question.
    – Jay
    Jul 21, 2014 at 13:39
  • 1
    A possible drawback of delete is that you sometimes might want to support multiple json views of your struct (map). For example json view for the client without a sensitive field, and json view for the database WITH the sensitive field. Fortunately it is still possible to use the struct -- just have a look at my answer. Jul 8, 2015 at 20:26
  • This works for me as I only needed a specific Id but, do not want to return the entire json struct. Thanks for this! Mar 4, 2020 at 9:34
  • A tag is just part of the reflection and that's how it's used to encode/decode JSON data. Could you change the reflection dynamically and in this way do what the OP wanted?
    – mandarin
    Feb 24, 2021 at 16:06
236

use `json:"-"`

// Field is ignored by this package.
Field int `json:"-"`

// Field appears in JSON as key "myName".
Field int `json:"myName"`

// Field appears in JSON as key "myName" and
// the field is omitted from the object if its value is empty,
// as defined above.
Field int `json:"myName,omitempty"`

// Field appears in JSON as key "Field" (the default), but
// the field is skipped if empty.
// Note the leading comma.
Field int `json:",omitempty"`

doc : http://golang.org/pkg/encoding/json/#Marshal

3
  • 20
    I'd disagree @Jacob because the OP said they wanted to dynamically control the output fields based on query string entries to the API. For example, if the caller to the API only asks for Industry and Country, you would then need to remove the rest. This is why the "ticked" answer is marked as an answer to this question. This highly-voted answer is for marking fields explicitly never-available-to-any-builtin-json-marshaler - EVER. if you want it dynamically, the ticked answer is the answer.
    – eduncan911
    Nov 18, 2014 at 19:18
  • 24
    This is the answer most people ending up here from searching would want, but it's not the answer to the question. Jan 16, 2015 at 17:48
  • 5
    As stated already the OP was asking for a method to dynamically form a DTO.
    – user986408
    May 7, 2015 at 20:57
82

Another way to do this is to have a struct of pointers with the ,omitempty tag. If the pointers are nil, the fields won't be Marshalled.

This method will not require additional reflection or inefficient use of maps.

Same example as jorelli using this method: http://play.golang.org/p/JJNa0m2_nw

5
  • 3
    +1 Completely agree. I use this rule/trick all the time with the built-in marshalers (and even built a CSV reader/writer based off of this rule as well! - I may open-source that soon as yet another csv go package). The OP could then simply not set the *Country value to nil, and it would be omitted. And awesome that you supplied a nice;y typed play.golang as well.
    – eduncan911
    Nov 18, 2014 at 19:30
  • 2
    Of course that method requires reflection, the stdlib's json-to-struct marshaling always uses reflection (actually it always uses reflection period, map or struct or whatever).
    – mna
    Mar 29, 2015 at 18:52
  • 2
    Yes, but it does not require additional reflection using interfaces, which some other answers recommend.
    – Druska
    Mar 30, 2015 at 17:07
  • 1
    this is the best way to dynamically display fields Aug 15, 2022 at 14:58
  • 1
    This is the best answer imo. Jun 28 at 1:50
20

You can use the reflect package to select the fields that you want by reflecting on the field tags and selecting the json tag values. Define a method on your SearchResults type that selects the fields you want and returns them as a map[string]interface{}, and then marshal that instead of the SearchResults struct itself. Here's an example of how you might define that method:

func fieldSet(fields ...string) map[string]bool {
    set := make(map[string]bool, len(fields))
    for _, s := range fields {
        set[s] = true
    }
    return set
}

func (s *SearchResult) SelectFields(fields ...string) map[string]interface{} {
    fs := fieldSet(fields...)
    rt, rv := reflect.TypeOf(*s), reflect.ValueOf(*s)
    out := make(map[string]interface{}, rt.NumField())
    for i := 0; i < rt.NumField(); i++ {
        field := rt.Field(i)
        jsonKey := field.Tag.Get("json")
        if fs[jsonKey] {
            out[jsonKey] = rv.Field(i).Interface()
        }
    }
    return out
}

and here's a runnable solution that shows how you would call this method and marshal your selection: http://play.golang.org/p/1K9xjQRnO8

2
  • come to think of it, you could reasonably generalize the selectfields pattern to any type and any tag key; there's nothing about this that is specific to the SearchResult definition or the json key.
    – jorelli
    Jun 26, 2013 at 14:51
  • 1
    I'm trying to stay away from reflection but this saves type information pretty nicely... It's nice to have code that documents what your structures look like better than a bunch of if/else tags in a validate() method (if you even have one)
    – Aktau
    Oct 9, 2013 at 13:40
9

I just published sheriff, which transforms structs to a map based on tags annotated on the struct fields. You can then marshal (JSON or others) the generated map. It probably doesn't allow you to only serialize the set of fields the caller requested, but I imagine using a set of groups would allow you to cover most cases. Using groups instead of the fields directly would most likely also increase cache-ability.

Example:

package main

import (
    "encoding/json"
    "fmt"
    "log"

    "github.com/hashicorp/go-version"
    "github.com/liip/sheriff"
)

type User struct {
    Username string   `json:"username" groups:"api"`
    Email    string   `json:"email" groups:"personal"`
    Name     string   `json:"name" groups:"api"`
    Roles    []string `json:"roles" groups:"api" since:"2"`
}

func main() {
    user := User{
        Username: "alice",
        Email:    "[email protected]",
        Name:     "Alice",
        Roles:    []string{"user", "admin"},
    }

    v2, err := version.NewVersion("2.0.0")
    if err != nil {
        log.Panic(err)
    }

    o := &sheriff.Options{
        Groups:     []string{"api"},
        ApiVersion: v2,
    }

    data, err := sheriff.Marshal(o, user)
    if err != nil {
        log.Panic(err)
    }

    output, err := json.MarshalIndent(data, "", "  ")
    if err != nil {
        log.Panic(err)
    }
    fmt.Printf("%s", output)
}
0
9

Take three ingredients:

  1. The reflect package to loop over all the fields of a struct.

  2. An if statement to pick up the fields you want to Marshal, and

  3. The encoding/json package to Marshal the fields of your liking.

Preparation:

  1. Blend them in a good proportion. Use reflect.TypeOf(your_struct).Field(i).Name() to get a name of the ith field of your_struct.

  2. Use reflect.ValueOf(your_struct).Field(i) to get a type Value representation of an ith field of your_struct.

  3. Use fieldValue.Interface() to retrieve the actual value (upcasted to type interface{}) of the fieldValue of type Value (note the bracket use - the Interface() method produces interface{}

If you luckily manage not to burn any transistors or circuit-breakers in the process you should get something like this:

func MarshalOnlyFields(structa interface{},
    includeFields map[string]bool) (jsona []byte, status error) {
    value := reflect.ValueOf(structa)
    typa := reflect.TypeOf(structa)
    size := value.NumField()
    jsona = append(jsona, '{')
    for i := 0; i < size; i++ {
        structValue := value.Field(i)
        var fieldName string = typa.Field(i).Name
        if marshalledField, marshalStatus := json.Marshal((structValue).Interface()); marshalStatus != nil {
            return []byte{}, marshalStatus
        } else {
            if includeFields[fieldName] {
                jsona = append(jsona, '"')
                jsona = append(jsona, []byte(fieldName)...)
                jsona = append(jsona, '"')
                jsona = append(jsona, ':')
                jsona = append(jsona, (marshalledField)...)
                if i+1 != len(includeFields) {
                    jsona = append(jsona, ',')
                }
            }
        }
    }
    jsona = append(jsona, '}')
    return
}

Serving:

serve with an arbitrary struct and a map[string]bool of fields you want to include, for example

type magic struct {
    Magic1 int
    Magic2 string
    Magic3 [2]int
}

func main() {
    var magic = magic{0, "tusia", [2]int{0, 1}}
    if json, status := MarshalOnlyFields(magic, map[string]bool{"Magic1": true}); status != nil {
        println("error")
    } else {
        fmt.Println(string(json))
    }

}

Bon Appetit!

1
  • 1
    Warning! If your includeFields contain field names, which do not match the actual fields, you're going to get an invalid json. You have been warned. Jul 8, 2015 at 21:27
8

I created this function to convert struct to JSON string by ignoring some fields. Hope it will help.

func GetJSONString(obj interface{}, ignoreFields ...string) (string, error) {
    toJson, err := json.Marshal(obj)
    if err != nil {
        return "", err
    }

    if len(ignoreFields) == 0 {
        return string(toJson), nil
    }

    toMap := map[string]interface{}{}
    json.Unmarshal([]byte(string(toJson)), &toMap)

    for _, field := range ignoreFields {
        delete(toMap, field)
    }

    toJson, err = json.Marshal(toMap)
    if err != nil {
        return "", err
    }
    return string(toJson), nil
}

Example: https://play.golang.org/p/nmq7MFF47Gp

1
  • 2
    Kudos for not over-engineering the living jeebers out of the issue. You probably don't need to do the marshal/map/marshal business, but nice solution!
    – Sam Hughes
    Jun 23, 2021 at 6:21
5

You can use tagging attribute "omitifempty" or make optional fields pointers and leave those you want skipped uninitialized.

2
  • This is the most correct answer to the OPs question and use case. Mar 1, 2015 at 7:31
  • 4
    @user1943442, not it is not; the OP explicitly mentions why "omitempty" is inapplicable.
    – Dave C
    Mar 29, 2015 at 19:00
5

Here is how I defined my structure.

type User struct {
    Username string  `json:"username" bson:"username"`
    Email    string  `json:"email" bson:"email"`
    Password *string `json:"password,omitempty" bson:"password"`
    FullName string  `json:"fullname" bson:"fullname"`
}

And inside my function set user.Password = nil for not to be Marshalled.

3

The question is now a bit old, but I came across the same issue a little while ago, and as I found no easy way to do this, I built a library fulfilling this purpose. It allows to easily generate a map[string]interface{} from a static struct.

https://github.com/tuvistavie/structomap

2
  • You can now easily do it using a code snippet from my recipe. Jul 8, 2015 at 21:34
  • The snippet is a subset of the library, but a major issue here about returning a []byte is that it is not very reusable: no easy way to add a field afterwards, for example. So I would suggest to create a map[string]interface{} and let the JSON serialization part to the standard library. Jul 30, 2015 at 6:31
3

I didn't have the same problem but similar. Below code solves your problem too, of course if you don't mind performance issue. Before implement that kind of solution to your system I recommend you to redesign your structure if you can. Sending variable structure response is over-engineering. I believe a response structure represents a contract between a request and resource and it should't be depend requests.(you can make un-wanted fields null, I do). In some cases we have to implement this design, if you believe you are in that cases here is the play link and code I use.

type User2 struct {
    ID       int    `groups:"id" json:"id,omitempty"`
    Username string `groups:"username" json:"username,omitempty"`
    Nickname string `groups:"nickname" json:"nickname,omitempty"`
}

type User struct {
    ID       int    `groups:"private,public" json:"id,omitempty"`
    Username string `groups:"private" json:"username,omitempty"`
    Nickname string `groups:"public" json:"nickname,omitempty"`
}

var (
    tagName = "groups"
)

//OmitFields sets fields nil by checking their tag group value and access control tags(acTags)
func OmitFields(obj interface{}, acTags []string) {
    //nilV := reflect.Value{}
    sv := reflect.ValueOf(obj).Elem()
    st := sv.Type()
    if sv.Kind() == reflect.Struct {
        for i := 0; i < st.NumField(); i++ {
            fieldVal := sv.Field(i)
            if fieldVal.CanSet() {
                tagStr := st.Field(i).Tag.Get(tagName)
                if len(tagStr) == 0 {
                    continue
                }
                tagList := strings.Split(strings.Replace(tagStr, " ", "", -1), ",")
                //fmt.Println(tagList)
                // ContainsCommonItem checks whether there is at least one common item in arrays
                if !ContainsCommonItem(tagList, acTags) {
                    fieldVal.Set(reflect.Zero(fieldVal.Type()))
                }
            }
        }
    }
}

//ContainsCommonItem checks if arrays have at least one equal item
func ContainsCommonItem(arr1 []string, arr2 []string) bool {
    for i := 0; i < len(arr1); i++ {
        for j := 0; j < len(arr2); j++ {
            if arr1[i] == arr2[j] {
                return true
            }
        }
    }
    return false
}
func main() {
    u := User{ID: 1, Username: "very secret", Nickname: "hinzir"}
    //assume authenticated user doesn't has permission to access private fields
    OmitFields(&u, []string{"public"}) 
    bytes, _ := json.Marshal(&u)
    fmt.Println(string(bytes))


    u2 := User2{ID: 1, Username: "very secret", Nickname: "hinzir"}
    //you want to filter fields by field names
    OmitFields(&u2, []string{"id", "nickname"}) 
    bytes, _ = json.Marshal(&u2)
    fmt.Println(string(bytes))

}
1
  • reflect is slow
    – Eric
    Sep 3, 2022 at 4:27
3

I also faced this problem, at first I just wanted to specialize the responses in my http handler. My first approach was creating a package that copies the information of a struct to another struct and then marshal that second struct. I did that package using reflection, so, never liked that approach and also I wasn't dynamically.

So I decided to modify the encoding/json package to do this. The functions Marshal, MarshalIndent and (Encoder) Encode additionally receives a

type F map[string]F

I wanted to simulate a JSON of the fields that are needed to marshal, so it only marshals the fields that are in the map.

https://github.com/jtorz/jsont

package main

import (
    "fmt"
    "log"
    "net/http"

    "github.com/jtorz/jsont/v2"
)

type SearchResult struct {
    Date        string      `json:"date"`
    IdCompany   int         `json:"idCompany"`
    Company     string      `json:"company"`
    IdIndustry  interface{} `json:"idIndustry"`
    Industry    string      `json:"industry"`
    IdContinent interface{} `json:"idContinent"`
    Continent   string      `json:"continent"`
    IdCountry   interface{} `json:"idCountry"`
    Country     string      `json:"country"`
    IdState     interface{} `json:"idState"`
    State       string      `json:"state"`
    IdCity      interface{} `json:"idCity"`
    City        string      `json:"city"`
} //SearchResult

type SearchResults struct {
    NumberResults int            `json:"numberResults"`
    Results       []SearchResult `json:"results"`
} //type SearchResults
func main() {
    msg := SearchResults{
        NumberResults: 2,
        Results: []SearchResult{
            {
                Date:        "12-12-12",
                IdCompany:   1,
                Company:     "alfa",
                IdIndustry:  1,
                Industry:    "IT",
                IdContinent: 1,
                Continent:   "america",
                IdCountry:   1,
                Country:     "México",
                IdState:     1,
                State:       "CDMX",
                IdCity:      1,
                City:        "Atz",
            },
            {
                Date:        "12-12-12",
                IdCompany:   2,
                Company:     "beta",
                IdIndustry:  1,
                Industry:    "IT",
                IdContinent: 1,
                Continent:   "america",
                IdCountry:   2,
                Country:     "USA",
                IdState:     2,
                State:       "TX",
                IdCity:      2,
                City:        "XYZ",
            },
        },
    }
    fmt.Println(msg)
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {

        //{"numberResults":2,"results":[{"date":"12-12-12","idCompany":1,"idIndustry":1,"country":"México"},{"date":"12-12-12","idCompany":2,"idIndustry":1,"country":"USA"}]}
        err := jsont.NewEncoder(w).Encode(msg, jsont.F{
            "numberResults": nil,
            "results": jsont.F{
                "date":       nil,
                "idCompany":  nil,
                "idIndustry": nil,
                "country":    nil,
            },
        })
        if err != nil {
            log.Fatal(err)
        }
    })

    http.ListenAndServe(":3009", nil)
}
1
  • I haven't tried it yet, but this looks great. It would be even better if Marshaler interface is also supported as well.
    – huggie
    Feb 12, 2020 at 7:08
1

Update/remove struct fields from serialization based on another field in that struct dynamically

AKA: Don't serialize the `Content` field if the `Hidden` field is true. Or, Set `Content` to "nothing found!" if the `Hidden` field is true.


Example:

For this struct (where CustomStruct is any custom struct you have):

type Post struct {
    Hidden bool `json:"hidden"`
    Content string `json:"content"`
    Value CustomStruct `json:"value"`
}

You can choose to dynamically return or not the value of any field based on the bool value of Hidden.

First, create an alias for your struct:

type Alias Post

Then make a wrapper for the alias:

type AuxPost struct {
    *Alias
}

Then override the MarshalJSON method (importing package "encoding/json"):

func (p Post) MarshalJSON() ([]byte, error) {
    type Alias Post
    if p.Hidden {
        hiddenPost := struct {
            Alias
            Content string `json:"content"`
            Value   bool   `json:"value,omitempty"`
        }{
            Alias:   (Alias)(p),
            Content: "[this post was removed]",
            Value:   false,
        }
        return json.Marshal(hiddenPost)

    }
    regularPost := struct {
        Alias
    }{
        Alias: (Alias)(p),
    }
    return json.Marshal(regularPost)
}

If your post's Hidden field is true, then it will serialize normally. If not, the Content field will be set to "[this post was removed]" and the Value field will be completely omitted (because in this particular case I set omitempty).

... the case where Hidden is false serializes as:

{
  "hidden": true,
  "content": "hello world",
  "value": {
    "note": "your custom struct can have anything"
  }
}

... and the case where Hidden is true serializes as:

{
  "hidden": true,
  "content": "[this post was removed]"
}

Removing a non-struct field is trivial, but the way the struct getting removed works is by tricking it into thinking its a boolean, and then setting that bool to false, and omitting zero values from serializations (bool's zero-value = false).

EDIT: If we wanted the Value struct to just be serialized as null instead, we got remove the omitempty tag and instead set its type (previously bool) to *interface{} and then below as nil (previously false).

0

To extend chhaileng answer, here is the version that remove all occurrences of a field with recursion

// GetJSONWithOutFields - Description: return a string representation of an interface with specified fields removed
func GetJSONWithOutFields(obj interface{}, ignoreFields ...string) (string, error) {
    toJson, err := json.Marshal(obj)
    if err != nil {
        return "", err
    }

    if len(ignoreFields) == 0 {
        return string(toJson), nil
    }

    toMap := map[string]interface{}{}
    err = json.Unmarshal(toJson, &toMap)
    if err != nil {
        return "", err
    }

    for _, field := range ignoreFields {
        DeleteField(toMap, field)
    }

    toJson, err = json.Marshal(toMap)
    if err != nil {
        return "", err
    }
    return string(toJson), nil
}

// DeleteField - Description: recursively delete field
func DeleteField(toMap map[string]interface{}, field string) {
    delete(toMap, field)
    for _, v := range toMap {
        if m, isMap := v.(map[string]interface{}); isMap {
            DeleteField(m, field)
        }
    }
}
0

Use pointers with omitempty:

package main

import (
    "encoding/json"
    "fmt"
)

type Foo struct {
    N int    `json:"n,omitempty"`
    S string `json:"s,omitempty"`
}

type Bar struct {
    N *int    `json:"n,omitempty"`
    S *string `json:"s,omitempty"`
}

func main() {
    var zero int
    var empty string

    foo := Foo{
        N: zero,
        S: empty,
    }
    data, _ := json.Marshal(foo)
    fmt.Println(string(data)) // prints {}

    bar := Bar{
        N: &zero,
        S: &empty,
    }
    data, _ = json.Marshal(bar)
    fmt.Println(string(data)) // prints {"n":0,"s":""}
}

omitempty omits zero values, and pointer to something is not a zero value of pointer, even if variable pointed has zero value. To omit value from JSON set the pointer to nil (or just don't initialize it)

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.