Let's Go Further Filtering, sorting, and pagination › Returning pagination metadata
Previous · Contents · Next
Chapter 9.8.

Returning pagination metadata

At this point, the pagination on our GET /v1/movies endpoint is working nicely, but it would be even better if we could include some additional metadata along with the response. Information like the current and last page numbers, and the total number of available records would help give the client context about the response and make navigating through the pages easier.

In this chapter we’ll improve the response so that it includes additional pagination metadata, similar to this:

{
    "metadata": {
        "current_page": 1,
        "page_size": 20,
        "first_page": 1,
        "last_page": 42,
        "total_records": 832
    },
    "movies": [
        {
            "id": 1,
            "title": "Moana",
            "year": 2015,
            "runtime": "107 mins",
            "genres": [
                "animation",
                "adventure"
            ],
            "version": 1
        },
        ...
    ]
}

Calculating the total records

The challenging part of doing this is generating the total_records figure. We want this to reflect the total number of available records given the title and genres filters that are applied — not the absolute total of records in the movies table.

A neat way to do this is to adapt our existing SQL query to include a window function which counts the total number of filtered rows, like so:

SELECT count(*) OVER(), id, created_at, title, year, runtime, genres, version
FROM movies
WHERE (to_tsvector('simple', title) @@ plainto_tsquery('simple', $1) OR $1 = '') 
AND (genres @> $2 OR $2 = '{}')     
ORDER BY %s %s, id ASC
LIMIT $3 OFFSET $4

The inclusion of the count(*) OVER() expression at the start of the query will result in the filtered record count being included as the first value in each row. A bit like this:

 count | id |       created_at       |       title        | year | runtime |          genres           | version 
-------+----+------------------------+--------------------+------+---------+---------------------------+---------
     3 |  1 | 2020-11-27 17:17:25+01 | Moana              | 2015 |     107 | {animation,adventure}     |       1
     3 |  2 | 2020-11-27 18:01:45+01 | Black Panther      | 2018 |     134 | {sci-fi,action,adventure} |       2
     3 |  4 | 2020-11-27 18:02:20+01 | The Breakfast Club | 1985 |      97 | {comedy,drama}            |       6

When PostgreSQL executes this SQL query, the (very simplified) sequence of events runs broadly like this:

  1. The WHERE clause is used to filter the data in the movies table and get the qualifying rows.
  2. The window function count(*) OVER() is applied, which counts all the qualifying rows.
  3. The ORDER BY rules are applied and the qualifying rows are sorted.
  4. The LIMIT and OFFSET rules are applied and the appropriate sub-set of sorted qualifying rows is returned.

Updating the code

With that brief explanation out of the way, let’s get this up and running. We’ll begin by updating the internal/data/filters.go file to define a new Metadata struct to hold the pagination metadata, along with a helper to calculate the values. Like so:

File: internal/data/filters.go
package data

import (
    "math" // New import
    "strings"

    "greenlight.alexedwards.net/internal/validator"
)

...

// Define a new Metadata struct for holding the pagination metadata.
type Metadata struct {
    CurrentPage  int `json:"current_page,omitzero"`
    PageSize     int `json:"page_size,omitzero"`
    FirstPage    int `json:"first_page,omitzero"`
    LastPage     int `json:"last_page,omitzero"`
    TotalRecords int `json:"total_records,omitzero"`
}

// The calculateMetadata() function calculates the appropriate pagination metadata 
// values given the total number of records, current page, and page size values. Note 
// that when the last page value is calculated we are dividing two int values, and 
// when dividing integer types in Go the result will also be an integer type, with 
// the modulus (or remainder) dropped. So, for example, if there were 12 records in total 
// and a page size of 5, the last page value would be (12+5-1)/5 = 3.2, which is then
// truncated to 3 by Go. 
func calculateMetadata(totalRecords, page, pageSize int) Metadata {
    if totalRecords == 0 {
        // Note that we return an empty Metadata struct if there are no records.
        return Metadata{}
    }

    return Metadata{
        CurrentPage:  page,
        PageSize:     pageSize,
        FirstPage:    1,
        LastPage:     (totalRecords + pageSize - 1) / pageSize,
        TotalRecords: totalRecords,
    }
}

Then we need to head back to our GetAll() method and update it to use our new SQL query (with the window function) to get the total records count. Then, if everything works successfully, we’ll use the calculateMetadata() function to generate the pagination metadata and return it along with the movie data.

Go ahead and update the GetAll() function like so:

File: internal/data/movies.go
package data

...

// Update the function signature to return a Metadata struct.
func (m MovieModel) GetAll(title string, genres []string, filters Filters) ([]*Movie, Metadata, error) {
    // Update the SQL query to include the window function which counts the total 
    // (filtered) records.
    query := fmt.Sprintf(`
        SELECT count(*) OVER(), id, created_at, title, year, runtime, genres, version
        FROM movies
        WHERE (to_tsvector('simple', title) @@ plainto_tsquery('simple', $1) OR $1 = '') 
        AND (genres @> $2 OR $2 = '{}')     
        ORDER BY %s %s, id ASC
        LIMIT $3 OFFSET $4`, filters.sortColumn(), filters.sortDirection())

    ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
    defer cancel()

    args := []any{title, pq.Array(genres), filters.limit(), filters.offset()}

    rows, err := m.DB.QueryContext(ctx, query, args...)
    if err != nil {
        return nil, Metadata{}, err // Update this to return an empty Metadata struct.
    }

    defer rows.Close()

    // Declare a totalRecords variable.
    totalRecords := 0
    movies := []*Movie{}

    for rows.Next() {
        var movie Movie

        err := rows.Scan(
            &totalRecords, // Scan the count from the window function into totalRecords.
            &movie.ID,
            &movie.CreatedAt,
            &movie.Title,
            &movie.Year,
            &movie.Runtime,
            pq.Array(&movie.Genres),
            &movie.Version,
        )
        if err != nil {
            return nil, Metadata{}, err // Update this to return an empty Metadata struct.
        }

        movies = append(movies, &movie)
    }

    if err = rows.Err(); err != nil {
        return nil, Metadata{}, err // Update this to return an empty Metadata struct.
    }

    // Generate a Metadata struct, passing in the total record count and pagination
    // parameters from the client.
    metadata := calculateMetadata(totalRecords, filters.Page, filters.PageSize)

    // Include the metadata struct when returning.
    return movies, metadata, nil
}

Finally, we need to update our listMoviesHandler handler to receive the Metadata struct returned by GetAll() and include the information in the JSON response. Like so:

File: cmd/api/movies.go
package main

...

func (app *application) listMoviesHandler(w http.ResponseWriter, r *http.Request) {
    var input struct {
        Title  string
        Genres []string
        data.Filters
    }

    v := validator.New()

    qs := r.URL.Query()

    input.Title = app.readString(qs, "title", "")
    input.Genres = app.readCSV(qs, "genres", []string{})

    input.Filters.Page = app.readInt(qs, "page", 1, v)
    input.Filters.PageSize = app.readInt(qs, "page_size", 20, v)

    input.Filters.Sort = app.readString(qs, "sort", "id")
    input.Filters.SortSafelist = []string{"id", "title", "year", "runtime", "-id", "-title", "-year", "-runtime"}

    if data.ValidateFilters(v, input.Filters); !v.Valid() {
        app.failedValidationResponse(w, r, v.Errors)
        return
    }

    // Accept the metadata struct as a return value.
    movies, metadata, err := app.models.Movies.GetAll(input.Title, input.Genres, input.Filters)
    if err != nil {
        app.serverErrorResponse(w, r, err)
        return
    }

    // Include the metadata in the response envelope.
    err = app.writeJSON(w, http.StatusOK, envelope{"movies": movies, "metadata": metadata}, nil)
    if err != nil {
        app.serverErrorResponse(w, r, err)
    }
}

Feel free to restart the API and try out this new functionality by making some different requests to the GET /v1/movies endpoint. You should find that the correct pagination metadata is now included in the response. For example:

$ curl "localhost:4000/v1/movies?page=1&page_size=2"
{
    "metadata": {
        "current_page": 1,
        "page_size": 2,
        "first_page": 1,
        "last_page": 2,
        "total_records": 3
    },
    "movies": [
        {
            "id": 1,
            "title": "Moana",
            "year": 2015,
            "runtime": "107 mins",
            "genres": [
                "animation",
                "adventure"
            ],
            "version": 1
        },
        {
            "id": 2,
            "title": "Black Panther",
            "year": 2018,
            "runtime": "134 mins",
            "genres": [
                "sci-fi",
                "action",
                "adventure"
            ],
            "version": 2
        }
    ]
}

And if you try making a request with a filter applied, you should see that the last_page value and total_records count changes to reflect the applied filters. For example, by requesting only movies with the genre “adventure” we can see that the total_records count drops down to 2:

$ curl "localhost:4000/v1/movies?genres=adventure"
{
    "metadata": {
        "current_page": 1,
        "page_size": 20,
        "first_page": 1,
        "last_page": 1,
        "total_records": 2
    },
    "movies": [
        {
            "id": 1,
            "title": "Moana",
            "year": 2015,
            "runtime": "107 mins",
            "genres": [
                "animation",
                "adventure"
            ],
            "version": 1
        },
        {
            "id": 2,
            "title": "Black Panther",
            "year": 2018,
            "runtime": "134 mins",
            "genres": [
                "sci-fi",
                "action",
                "adventure"
            ],
            "version": 2
        }
    ]
}

Lastly, if you make a request with a too-high page value, you should get a response with an empty metadata object and an empty movies array, like this:

$ curl "localhost:4000/v1/movies?page=100"
{
    "metadata": {},
    "movies": []
}

Over the last few chapters, we’ve had to put in a lot of work on the GET /v1/movies endpoint. But the end result is really powerful. The client now has a lot of control over what their response contains, with filtering, pagination and sorting all supported.

With the Filters struct that we’ve created, we’ve also got something that we can easily drop into any other endpoints that need pagination and sorting functionality in the future. And if you take a step back and look at the final code we’ve written in the listMoviesHandler and our database model’s GetAll() method, there isn’t so much more code than there was in the earlier iterations of the endpoint.