Validating JSON input
In many cases, you’ll want to perform additional validation checks on the data from a client to make sure it meets your specific business rules before processing it. In this chapter, we’ll illustrate how to do that in the context of a JSON API by updating our createMovieHandler to check that:
- The movie title provided by the client is not empty and is no more than 500 bytes long.
- The movie year is not empty and is between 1888 and the current year.
- The movie runtime is not empty and is a positive integer.
- The movie has between one and five (unique) genres.
If any of those checks fail, we want to send the client a 422 Unprocessable Entity response along with error messages which clearly describe the validation failures.
Creating a validator package
To help us with validation throughout this project, we’re going to create a small internal/validator package with some simple reusable helper types and functions. If you’re coding along, go ahead and create the following directory and file on your machine:
$ mkdir internal/validator $ touch internal/validator/validator.go
Then in this new internal/validator/validator.go file, add the following code:
package validator import ( "regexp" "slices" ) // Declare a regular expression for sanity checking the format of email addresses (we'll // use this later in the book). If you're interested, this regular expression pattern is // taken from https://html.spec.whatwg.org/#valid-e-mail-address. Note: if you're // reading this in PDF or EPUB format and cannot see the full pattern, please see the // note further down the page. var ( EmailRX = regexp.MustCompile("^[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$") ) // Define a new Validator type which contains a map of validation errors. type Validator struct { Errors map[string]string } // New is a helper which creates a new Validator instance with an empty errors map. func New() *Validator { return &Validator{Errors: make(map[string]string)} } // Valid returns true if the errors map doesn't contain any entries. func (v *Validator) Valid() bool { return len(v.Errors) == 0 } // AddError adds an error message to the map (so long as no entry already exists for // the given key). func (v *Validator) AddError(key, message string) { if _, exists := v.Errors[key]; !exists { v.Errors[key] = message } } // Check adds an error message to the map only if a validation check is not 'ok'. func (v *Validator) Check(ok bool, key, message string) { if !ok { v.AddError(key, message) } } // Generic function which returns true if a specific value is in a list of permitted // values. func PermittedValue[T comparable](value T, permittedValues ...T) bool { return slices.Contains(permittedValues, value) } // Matches returns true if a string value matches a specific regexp pattern. func Matches(value string, rx *regexp.Regexp) bool { return rx.MatchString(value) } // Generic function which returns true if all values in a slice are unique. func Unique[T comparable](values []T) bool { uniqueValues := make(map[T]bool) for _, value := range values { uniqueValues[value] = true } return len(values) == len(uniqueValues) }
To summarize this:
In the code above we’ve defined a custom Validator type which contains a map of errors. The Validator type provides a Check() method for conditionally adding errors to the map, and a Valid() method which returns whether the errors map is empty or not. We’ve also added PermittedValue(), Matches() and Unique() functions to help us perform some specific validation checks.
Conceptually this Validator type is quite basic, but that’s not a bad thing. As we’ll see over the course of this book, it’s surprisingly powerful in practice and gives us a lot of flexibility and control over validation checks and how we perform them.
If you’re reading this book in PDF or EPUB format and you can’t see the full EmailRX regexp pattern in the code snippet above, here it is broken up into multiple lines:
"^[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?
(?:\\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$"
In your code, this regexp pattern should all be on a single line with no whitespace.
Performing validation checks
Alright, let’s start putting the Validator type to use!
The first thing we need to do is update our cmd/api/errors.go file to include a new failedValidationResponse() helper, which writes a 422 Unprocessable Entity and the contents of the errors map from our new Validator type as a JSON response body.
package main ... // Note that the errors parameter here has the type map[string]string, which is exactly // the same as the errors map contained in our Validator type. func (app *application) failedValidationResponse(w http.ResponseWriter, r *http.Request, errors map[string]string) { app.errorResponse(w, r, http.StatusUnprocessableEntity, errors) }
Then once that’s done, head back to your createMovieHandler and update it to perform the necessary validation checks on the input struct. Like so:
package main import ( "fmt" "net/http" "time" "greenlight.alexedwards.net/internal/data" "greenlight.alexedwards.net/internal/validator" // New import ) func (app *application) createMovieHandler(w http.ResponseWriter, r *http.Request) { var input struct { Title string `json:"title"` Year int32 `json:"year"` Runtime data.Runtime `json:"runtime"` Genres []string `json:"genres"` } err := app.readJSON(w, r, &input) if err != nil { app.badRequestResponse(w, r, err) return } // Initialize a new Validator instance. v := validator.New() // Use the Check() method to execute our validation checks. This will add the // provided key and error message to the errors map if the check does not evaluate // to true. For example, in the first line here we "check that the title is not // equal to the empty string". In the second, we "check that the length of the title // is less than or equal to 500 bytes" and so on. v.Check(input.Title != "", "title", "must be provided") v.Check(len(input.Title) <= 500, "title", "must not be more than 500 bytes long") v.Check(input.Year != 0, "year", "must be provided") v.Check(input.Year >= 1888, "year", "must be greater than 1888") v.Check(input.Year <= int32(time.Now().Year()), "year", "must not be in the future") v.Check(input.Runtime != 0, "runtime", "must be provided") v.Check(input.Runtime > 0, "runtime", "must be a positive integer") v.Check(input.Genres != nil, "genres", "must be provided") v.Check(len(input.Genres) >= 1, "genres", "must contain at least 1 genre") v.Check(len(input.Genres) <= 5, "genres", "must not contain more than 5 genres") // Note that we're using the Unique helper in the line below to check that all // values in the input.Genres slice are unique. v.Check(validator.Unique(input.Genres), "genres", "must not contain duplicate values") // Use the Valid() method to see if any of the checks failed. If they did, then use // the failedValidationResponse() helper to send a response to the client, passing // in the v.Errors map. if !v.Valid() { app.failedValidationResponse(w, r, v.Errors) return } fmt.Fprintf(w, "%+v\n", input) } ...
With that done, we should be good to try this out. Restart the API, then go ahead and issue a request to the POST /v1/movies endpoint containing some invalid data. Similar to this:
$ BODY='{"title":"","year":1000,"runtime":"-123 mins","genres":["sci-fi","sci-fi"]}'
$ curl -i -d "$BODY" localhost:4000/v1/movies
HTTP/1.1 422 Unprocessable Entity
Content-Type: application/json
Date: Wed, 07 Apr 2021 10:33:57 GMT
Content-Length: 180
{
"error": {
"genres": "must not contain duplicate values",
"runtime": "must be a positive integer",
"title": "must be provided",
"year": "must be greater than 1888"
}
}
That’s looking great. Our validation checks are working to prevent the request from being executed successfully — and even better — the client is getting a well-formed JSON response with clear, informative, error messages for each problem.
You can also try sending a valid request, if you like. You should find that the checks pass successfully and the input struct is dumped in the HTTP response, just like before:
$ BODY='{"title":"Moana","year":2016,"runtime":"107 mins","genres":["animation","adventure"]}'
$ curl -i -d "$BODY" localhost:4000/v1/movies
HTTP/1.1 200 OK
Date: Wed, 07 Apr 2021 10:35:40 GMT
Content-Length: 65
Content-Type: text/plain; charset=utf-8
{Title:Moana Year:2016 Runtime:107 Genres:[animation adventure]}
Making validation rules reusable
In large projects, it’s likely that you’ll want to reuse some of the same validation checks in multiple places. In our case — for example — we’ll want to use many of these same checks later when a client edits the movie data.
To prevent duplication, we can collect the validation checks for a movie into a standalone ValidateMovie() function. In theory this function could live almost anywhere in our codebase — next to the handlers in the cmd/api/movies.go file, or possibly in the internal/validators package. But personally, I like to keep the validation checks close to the relevant domain type in the internal/data package.
If you’re following along, reopen the internal/data/movies.go file and add a ValidateMovie() function containing the checks like so:
package data import ( "time" "greenlight.alexedwards.net/internal/validator" // New import ) type Movie struct { ID int64 `json:"id"` CreatedAt time.Time `json:"-"` Title string `json:"title"` Year int32 `json:"year,omitzero"` Runtime Runtime `json:"runtime,omitzero"` Genres []string `json:"genres,omitzero"` Version int32 `json:"version"` } func ValidateMovie(v *validator.Validator, movie *Movie) { v.Check(movie.Title != "", "title", "must be provided") v.Check(len(movie.Title) <= 500, "title", "must not be more than 500 bytes long") v.Check(movie.Year != 0, "year", "must be provided") v.Check(movie.Year >= 1888, "year", "must be greater than 1888") v.Check(movie.Year <= int32(time.Now().Year()), "year", "must not be in the future") v.Check(movie.Runtime != 0, "runtime", "must be provided") v.Check(movie.Runtime > 0, "runtime", "must be a positive integer") v.Check(movie.Genres != nil, "genres", "must be provided") v.Check(len(movie.Genres) >= 1, "genres", "must contain at least 1 genre") v.Check(len(movie.Genres) <= 5, "genres", "must not contain more than 5 genres") v.Check(validator.Unique(movie.Genres), "genres", "must not contain duplicate values") }
Once that’s done, we need to head back to our createMovieHandler and update it to initialize a new Movie struct, copy across the data from our input struct, and then call this new validation function. Like so:
package main ... func (app *application) createMovieHandler(w http.ResponseWriter, r *http.Request) { var input struct { Title string `json:"title"` Year int32 `json:"year"` Runtime data.Runtime `json:"runtime"` Genres []string `json:"genres"` } err := app.readJSON(w, r, &input) if err != nil { app.badRequestResponse(w, r, err) return } // Copy the values from the input struct to a new Movie struct. movie := &data.Movie{ Title: input.Title, Year: input.Year, Runtime: input.Runtime, Genres: input.Genres, } // Initialize a new Validator. v := validator.New() // Call the ValidateMovie() function, and if any checks fail, return a response // containing the errors. if data.ValidateMovie(v, movie); !v.Valid() { app.failedValidationResponse(w, r, v.Errors) return } fmt.Fprintf(w, "%+v\n", input) } ...
When you’re looking at this code, there might be a couple of questions in your head.
Firstly, you might be wondering why we’re initializing the Validator instance in our handler and passing it to the ValidateMovie() function — rather than initializing it in ValidateMovie() and passing it back as a return value.
This is because as our application gets more complex we will need to call multiple validation helpers from our handlers, rather than just one like we are above. So initializing the Validator in the handler, and then passing it around, gives us more flexibility.
You might also be wondering why we’re decoding the JSON request into the input struct and then copying the data across, rather than just decoding into the Movie struct directly.
The problem with decoding directly into a Movie struct is that a client could provide the keys id and version in their JSON request, and the corresponding values would be decoded without any error into the ID and Version fields of the Movie struct — even though we don’t want them to be. We could check the necessary fields in the Movie struct after the event to make sure that they are empty, but that feels a bit hacky, and decoding into an intermediary struct (like we are in our handler) is a cleaner, simpler, and more robust approach — albeit a little bit verbose.
OK, with those explanations out of the way, you should be able to start the application again and things should work the same as before from the client’s perspective. If you make an invalid request, you should get a response containing the error messages similar to this:
$ BODY='{"title":"","year":1000,"runtime":"-123 mins","genres":["sci-fi","sci-fi"]}'
$ curl -i -d "$BODY" localhost:4000/v1/movies
HTTP/1.1 422 Unprocessable Entity
Content-Type: application/json
Date: Wed, 07 Apr 2021 10:51:00 GMT
Content-Length: 180
{
"error": {
"genres": "must not contain duplicate values",
"runtime": "must be a positive integer",
"title": "must be provided",
"year": "must be greater than 1888"
}
}
Feel free to play around with this, and try sending different values in the JSON until you’re happy that all the validation checks are working as expected.