320

So I have the following, which seems incredibly hacky, and I've been thinking to myself that Go has better designed libraries than this, but I can't find an example of Go handling a POST request of JSON data. They are all form POSTs.

Here is an example request: curl -X POST -d "{\"test\": \"that\"}" http://localhost:8082/test

And here is the code, with the logs embedded:

package main

import (
    "encoding/json"
    "log"
    "net/http"
)

type test_struct struct {
    Test string
}

func test(rw http.ResponseWriter, req *http.Request) {
    req.ParseForm()
    log.Println(req.Form)
    //LOG: map[{"test": "that"}:[]]
    var t test_struct
    for key, _ := range req.Form {
        log.Println(key)
        //LOG: {"test": "that"}
        err := json.Unmarshal([]byte(key), &t)
        if err != nil {
            log.Println(err.Error())
        }
    }
    log.Println(t.Test)
    //LOG: that
}

func main() {
    http.HandleFunc("/test", test)
    log.Fatal(http.ListenAndServe(":8082", nil))
}

There's got to be a better way, right? I'm just stumped in finding what the best practice could be.

(Go is also known as Golang to the search engines, and mentioned here so others can find it.)

2
  • 3
    if you use curl -X POST -H 'Content-Type: application/json' -d "{\"test\": \"that\"}", then req.Form["test"] should return "that"
    – Vinicius
    Dec 5, 2015 at 23:42
  • @Vinicius are there any proofs of this?
    – diralik
    Dec 15, 2017 at 21:41

8 Answers 8

464

Please use json.Decoder instead of json.Unmarshal.

func test(rw http.ResponseWriter, req *http.Request) {
    decoder := json.NewDecoder(req.Body)
    var t test_struct
    err := decoder.Decode(&t)
    if err != nil {
        panic(err)
    }
    log.Println(t.Test)
}
18
  • 110
    Could you please explain why?
    – Ryan Bigg
    Jul 29, 2013 at 2:41
  • 112
    For start, it looks like this can handle a stream rather than needing you to load it all into a buffer yourself. (I'm a different Joe BTW)
    – Joe
    Sep 5, 2013 at 15:14
  • 12
    I wonder how proper error handling would look like in this case. I don't think it's a good idea to panic on an invalid json.
    – user986408
    Nov 25, 2014 at 18:52
  • 20
    I don't think you need to defer req.Body.Close() From the docs: "The Server will close the request body. The ServeHTTP Handler does not need to." Also to answer @thisisnotabus, from the docs: "For server requests the Request Body is always non-nil but will return EOF immediately when no body is present" golang.org/pkg/net/http/#Request Mar 7, 2017 at 22:25
  • 31
    I would suggest not using json.Decoder. It is intended for streams of JSON objects, not a single object. It is not more efficient for a single JSON object since it reads the entire object into memory. It has a downside that if garbage is included after the object it will not complain. Depending on a few factors, json.Decoder may not fully read the body and the connection will be ineligible for reuse.
    – Kale B
    May 21, 2017 at 20:58
107

You need to read from req.Body. The ParseForm method is reading from the req.Body and then parsing it in standard HTTP encoded format. What you want is to read the body and parse it in JSON format.

Here's your code updated.

package main

import (
    "encoding/json"
    "log"
    "net/http"
    "io/ioutil"
)

type test_struct struct {
    Test string
}

func test(rw http.ResponseWriter, req *http.Request) {
    body, err := ioutil.ReadAll(req.Body)
    if err != nil {
        panic(err)
    }
    log.Println(string(body))
    var t test_struct
    err = json.Unmarshal(body, &t)
    if err != nil {
        panic(err)
    }
    log.Println(t.Test)
}

func main() {
    http.HandleFunc("/test", test)
    log.Fatal(http.ListenAndServe(":8082", nil))
}
5
  • Thanks! I see where I was going wrong now. If you call req.ParseForm(), which I was doing in earlier attempts of trying to solve this problem, before you try and read the req.Body, it seems to clear the body out and unexpected end of JSON input is thrown when you go to Unmarshal (at least in 1.0.2)
    – TomJ
    Mar 28, 2013 at 1:46
  • 1
    @Daniel: When I do curl -X POST -d "{\"tes\": \"that\"}" localhost:8082/test, log.Println(t.Test) returns empty. Why ? Or for that matter if post any other JSON, it returns empty
    – Somesh
    Oct 30, 2014 at 12:46
  • Your POST request is wrong. tes != test. Appreciate that was 5 years ago :/
    – Rambatino
    Feb 8, 2019 at 4:19
  • This is a nice simple example!
    – 15412s
    Jan 11, 2020 at 11:24
  • This is good advice, but to be clear, the answers referring to the use of json.NewDecoder(req.Body) are also correct.
    – Rich
    Mar 21, 2020 at 22:01
96

There are two reasons why json.Decoder should be preferred over json.Unmarshal - that are not addressed in the most popular answer from 2013:

  1. February 2018, go 1.10 introduced a new method json.Decoder.DisallowUnknownFields() which addresses the concern of detecting unwanted JSON-input
  2. req.Body is already an io.Reader. Reading its entire contents and then performing json.Unmarshal wastes resources if the stream was, say a 10MB block of invalid JSON. Parsing the request body, with json.Decoder, as it streams in would trigger an early parse error if invalid JSON was encountered. Processing I/O streams in realtime is the preferred go-way.

Addressing some of the user comments about detecting bad user input:

To enforce mandatory fields, and other sanitation checks, try:

d := json.NewDecoder(req.Body)
d.DisallowUnknownFields() // catch unwanted fields

// anonymous struct type: handy for one-time use
t := struct {
    Test *string `json:"test"` // pointer so we can test for field absence
}{}

err := d.Decode(&t)
if err != nil {
    // bad JSON or unrecognized json field
    http.Error(rw, err.Error(), http.StatusBadRequest)
    return
}

if t.Test == nil {
    http.Error(rw, "missing field 'test' from JSON object", http.StatusBadRequest)
    return
}

// optional extra check
if d.More() {
    http.Error(rw, "extraneous data after JSON object", http.StatusBadRequest)
    return
}

// got the input we expected: no more, no less
log.Println(*t.Test)

Playground

Typical output:

$ curl -X POST -d "{}" http://localhost:8082/strict_test

expected json field 'test'

$ curl -X POST -d "{\"Test\":\"maybe?\",\"Unwanted\":\"1\"}" http://localhost:8082/strict_test

json: unknown field "Unwanted"

$ curl -X POST -d "{\"Test\":\"oops\"}g4rB4g3@#$%^&*" http://localhost:8082/strict_test

extraneous data after JSON

$ curl -X POST -d "{\"Test\":\"Works\"}" http://localhost:8082/strict_test 

log: 2019/03/07 16:03:13 Works
4
  • 19
    Thank you for explaining opinions instead of just stating that something is bad Aug 21, 2019 at 12:58
  • do you know what it doesn't handle? i saw Test can be in json twice and it accepts 2nd occurrence
    – tooptoop4
    Dec 24, 2019 at 10:48
  • @tooptoop4 one would need to write a custom decoder to warn about duplicate fields - adding inefficiencies to the decoder - all to handle a scenario that would never happen. No standard JSON encoder would ever produce duplicate fields.
    – colm.anseo
    Jan 28, 2020 at 13:58
  • More() does not really do what you seem to expect it to do: your example will happily accept {}] as valid JSON Mar 19, 2022 at 22:04
81

I was driving myself crazy with this exact problem. My JSON Marshaller and Unmarshaller were not populating my Go struct. Then I found the solution at https://eager.io/blog/go-and-json:

"As with all structs in Go, it’s important to remember that only fields with a capital first letter are visible to external programs like the JSON Marshaller."

After that, my Marshaller and Unmarshaller worked perfectly!

5
  • 3
    Please include some snippets from the link. If it gets deprecated the examples will be lost.
    – 030
    Apr 22, 2020 at 16:22
  • 2
    This answer not really related to question nor the provided link, but just about Marshal/Unmarshal JSON.
    – Mamrezo
    Oct 7, 2020 at 6:30
  • 4
    I just wasted a bunch of time figuring out this exact same thing! Feb 18, 2021 at 0:03
  • 1
    Just wasted an hour figuring out the same thing Sep 20, 2021 at 22:30
  • 2
    Oh God, why....
    – Liglo App
    Oct 13, 2022 at 10:58
22

I found the following example from the docs really helpful (source here).

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log"
    "strings"
)

func main() {
    const jsonStream = `
        {"Name": "Ed", "Text": "Knock knock."}
        {"Name": "Sam", "Text": "Who's there?"}
        {"Name": "Ed", "Text": "Go fmt."}
        {"Name": "Sam", "Text": "Go fmt who?"}
        {"Name": "Ed", "Text": "Go fmt yourself!"}
    `
    type Message struct {
        Name, Text string
    }
    dec := json.NewDecoder(strings.NewReader(jsonStream))
    for {
        var m Message
        if err := dec.Decode(&m); err == io.EOF {
            break
        } else if err != nil {
            log.Fatal(err)
        }
        fmt.Printf("%s: %s\n", m.Name, m.Text)
    }
}

The key here being that the OP was looking to decode

type test_struct struct {
    Test string
}

...in which case we would drop the const jsonStream, and replace the Message struct with the test_struct:

func test(rw http.ResponseWriter, req *http.Request) {
    dec := json.NewDecoder(req.Body)
    for {
        var t test_struct
        if err := dec.Decode(&t); err == io.EOF {
            break
        } else if err != nil {
            log.Fatal(err)
        }
        log.Printf("%s\n", t.Test)
    }
}

Update: I would also add that this post provides some great data about responding with JSON as well. The author explains struct tags, which I was not aware of.

Since JSON does not normally look like {"Test": "test", "SomeKey": "SomeVal"}, but rather {"test": "test", "somekey": "some value"}, you can restructure your struct like this:

type test_struct struct {
    Test string `json:"test"`
    SomeKey string `json:"some-key"`
}

...and now your handler will parse JSON using "some-key" as opposed to "SomeKey" (which you will be using internally).

0
5

I like to define custom structs locally. So:

// my handler func
func addImage(w http.ResponseWriter, r *http.Request) {

    // define custom type
    type Input struct {
        Url        string  `json:"url"`
        Name       string  `json:"name"`
        Priority   int8    `json:"priority"`
    }

    // define a var 
    var input Input

    // decode input or return error
    err := json.NewDecoder(r.Body).Decode(&input)
    if err != nil {
        w.WriteHeader(400)
        fmt.Fprintf(w, "Decode error! please check your JSON formating.")
        return
    }

    // print user inputs
    fmt.Fprintf(w, "Inputed name: %s", input.Name)

}
2
  • What if the body can come in a number of shapes, how do you first detect the shape and then use a strongly typed struct?
    – Dominic
    Dec 10, 2021 at 14:36
  • @dominic I guess the only way would be to declare a large struct Mar 1, 2022 at 20:04
4
type test struct {
    Test string `json:"test"`
}

func test(w http.ResponseWriter, req *http.Request) {
    var t test_struct

    body, _ := ioutil.ReadAll(req.Body)
    json.Unmarshal(body, &t)

    fmt.Println(t)
}
0

Earlier ReadAll func was part of ioutil package, but currently it got deprecated. But now the io package itself has the ReadAll func.

type test struct {
  Test string `json:"test"`
}

func test(w http.ResponseWriter, req *http.Request) {
  var t test_struct
  body, _ := io.ReadAll(req.Body)
  json.Unmarshal(body, &t)
  fmt.Println(t)
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.