Restricting Inputs
The changes that we made in the previous chapter to deal with invalid JSON and other bad requests were a big step in the right direction. But there are still a few things we can do to make our JSON processing even more robust.
One such thing is dealing with unknown fields. For example, you can try sending a request containing the unknown field rating to our createMovieHandler, like so:
$ curl -i -d '{"title": "Moana", "rating":"PG"}' localhost:4000/v1/movies
HTTP/1.1 200 OK
Date: Tue, 06 Apr 2021 18:51:50 GMT
Content-Length: 41
Content-Type: text/plain; charset=utf-8
{Title:Moana Year:0 Runtime:0 Genres:[]}
Notice how this request works without any problems — there’s no error to inform the client that the rating field is not recognized by our application. In certain scenarios, silently ignoring unknown fields may be exactly the behavior you want, but in our case it would be better if we could alert the client to the issue.
Fortunately, Go’s json.Decoder provides a DisallowUnknownFields() setting that we can use to generate an error when this happens.
Another problem we have is the fact that json.Decoder is designed to support streams of JSON data. When we call Decode() on our request body, it actually reads the first JSON value only from the body and decodes it. If we made a second call to Decode(), it would read and decode the second value and so on.
But because we call Decode() once — and only once — in our readJSON() helper, anything after the first JSON value in the request body is ignored. This means you could send a request body containing multiple JSON values, or garbage content after the first JSON value, and our API handlers would not raise an error. For example:
# Body contains multiple JSON values
$ curl -i -d '{"title": "Moana"}{"title": "Top Gun"}' localhost:4000/v1/movies
HTTP/1.1 200 OK
Date: Tue, 06 Apr 2021 18:53:57 GMT
Content-Length: 41
Content-Type: text/plain; charset=utf-8
{Title:Moana Year:0 Runtime:0 Genres:[]}
# Body contains garbage content after the first JSON value
$ curl -i -d '{"title": "Moana"} :~()' localhost:4000/v1/movies
HTTP/1.1 200 OK
Date: Tue, 06 Apr 2021 18:54:15 GMT
Content-Length: 41
Content-Type: text/plain; charset=utf-8
{Title:Moana Year:0 Runtime:0 Genres:[]}
Again, this behavior can be very useful, but it’s not the right fit for our use case. We want requests to our createMovieHandler handler to contain only a single JSON object in the request body, with information about the movie to be created in our system.
To ensure that there are no additional JSON values (or any other content) in the request body, we will need to call Decode() a second time in our readJSON() helper and check that it returns an io.EOF (end of file) error.
Finally, there’s currently no upper limit on the maximum size of the request body. This means that our createMovieHandler would be a good target for any malicious clients that wish to perform a denial-of-service attack on our API. We can address this by using the http.MaxBytesReader() function to limit the maximum size of the request body.
Let’s update our readJSON() helper to fix these three things:
package main import ( "encoding/json" "errors" "fmt" "io" "net/http" "strconv" "strings" // New import "github.com/julienschmidt/httprouter" ) ... func (app *application) readJSON(w http.ResponseWriter, r *http.Request, dst any) error { // Use http.MaxBytesReader() to limit the size of the request body to 1,048,576 // bytes (1MB). r.Body = http.MaxBytesReader(w, r.Body, 1_048_576) // Initialize the json.Decoder, and call the DisallowUnknownFields() method on it // before decoding. This means that if the JSON from the client now includes any // field that cannot be mapped to the target destination, the decoder will return // an error instead of just ignoring the field. dec := json.NewDecoder(r.Body) dec.DisallowUnknownFields() // Decode the request body to the destination. err := dec.Decode(dst) if err != nil { var syntaxError *json.SyntaxError var unmarshalTypeError *json.UnmarshalTypeError var invalidUnmarshalError *json.InvalidUnmarshalError // Add a new maxBytesError variable. var maxBytesError *http.MaxBytesError switch { case errors.As(err, &syntaxError): return fmt.Errorf("body contains badly-formed JSON (at character %d)", syntaxError.Offset) case errors.Is(err, io.ErrUnexpectedEOF): return errors.New("body contains badly-formed JSON") case errors.As(err, &unmarshalTypeError): if unmarshalTypeError.Field != "" { return fmt.Errorf("body contains incorrect JSON type for field %q", unmarshalTypeError.Field) } return fmt.Errorf("body contains incorrect JSON type (at character %d)", unmarshalTypeError.Offset) case errors.Is(err, io.EOF): return errors.New("body must not be empty") // If the JSON contains a field which cannot be mapped to the target destination // then Decode() will now return an error message in the format "json: unknown // field "<name>"". We check for this, extract the field name from the error, // and interpolate it into our custom error message. Note that there's an open // issue at https://github.com/golang/go/issues/29035 regarding turning this // into a distinct error type in the future. case strings.HasPrefix(err.Error(), "json: unknown field "): fieldName := strings.TrimPrefix(err.Error(), "json: unknown field ") return fmt.Errorf("body contains unknown key %s", fieldName) // Use the errors.As() function to check whether the error has the type // *http.MaxBytesError. If it does, then it means the request body exceeded our // size limit of 1MB and we return a clear error message. case errors.As(err, &maxBytesError): return fmt.Errorf("body must not be larger than %d bytes", maxBytesError.Limit) case errors.As(err, &invalidUnmarshalError): panic(err) default: return err } } // Call Decode() again, using a pointer to an empty anonymous struct as the // destination. If the request body only contained a single JSON value this will // return an io.EOF error. So if we get anything else, we know that there is // additional data in the request body and we return our own custom error message. err = dec.Decode(&struct{}{}) if !errors.Is(err, io.EOF) { return errors.New("body must only contain a single JSON value") } return nil }
Once you’ve made those changes, let’s try the requests from earlier in the chapter again:
$ curl -d '{"title": "Moana", "rating":"PG"}' localhost:4000/v1/movies
{
"error": "body contains unknown key \"rating\""
}
$ curl -d '{"title": "Moana"}{"title": "Top Gun"}' localhost:4000/v1/movies
{
"error": "body must only contain a single JSON value"
}
$ curl -d '{"title": "Moana"} :~()' localhost:4000/v1/movies
{
"error": "body must only contain a single JSON value"
}
Those are working much better now — processing of the request is terminated and the client receives a clear error message explaining exactly what the problem is.
Lastly, let’s try making a request with a very large JSON body.
To demonstrate this, I’ve created a 1.5MB JSON file that you can download into your /tmp directory by running the following command:
$ wget -O /tmp/largefile.json https://www.alexedwards.net/static/largefile.json
If you try making a request to your POST /v1/movies endpoint with this file as the request body, the http.MaxBytesReader() check will kick in and you should get a response similar to this:
$ curl -d @/tmp/largefile.json localhost:4000/v1/movies
{
"error": "body must not be larger than 1048576 bytes"
}
And with that, you’ll be glad to know that we’re finally finished with the readJSON() helper 😊
I must admit that the code inside readJSON() isn’t the most beautiful-looking… there’s a lot of error handling and logic that we’ve introduced for what is ultimately a one-line call to Decode(). But now it’s written, it’s done. You don’t need to touch it again, and it’s something that you can copy-and-paste into other projects easily.