Executing the shutdown

Chapter 11.3.

Intercepting the signals is all well and good, but it’s not very useful until we do something with them! In this chapter, we’re going to update our application so that the SIGINT and SIGTERM signals we intercept trigger a graceful shutdown of our API.

Specifically, after receiving one of these signals we will call the Shutdown() method on our HTTP server. The official documentation describes this as follows:

Shutdown gracefully shuts down the server without interrupting any active connections. Shutdown works by first closing all open listeners, then closing all idle connections, and then waiting indefinitely for connections to return to idle and then shut down.

The pattern to implement this in practice is difficult to describe with words, so let’s jump into the code and talk through the details as we go along.

File: cmd/api/server.go

package main

import (
    "context" // New import
    "errors"  // New import
    "fmt"
    "log/slog"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func (app *application) serve() error {
    srv := &http.Server{
        Addr:         fmt.Sprintf(":%d", app.config.port),
        Handler:      app.routes(),
        IdleTimeout:  time.Minute,
        ReadTimeout:  5 * time.Second,
        WriteTimeout: 10 * time.Second,
        ErrorLog:     slog.NewLogLogger(app.logger.Handler(), slog.LevelError),
    }

    // Create a shutdownError channel. We will use this to receive any errors returned
    // by the graceful Shutdown() function.
    shutdownError := make(chan error)

    go func() {
        // Intercept the signals, as before.
        quit := make(chan os.Signal, 1)
        signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
        s := <-quit

        // Update the log entry to read "shutting down server" instead of "caught signal".
        app.logger.Info("shutting down server", "signal", s.String())

        // Create a context with a 30-second timeout.
        ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
        defer cancel()

        // Call Shutdown() on our server, passing in the context we just made.
        // Shutdown() will return nil if the graceful shutdown was successful, or an
        // error (which may happen because of a problem closing the listeners, or 
        // because the shutdown didn't complete before the 30-second context deadline is
        // hit). We relay this return value to the shutdownError channel.
        shutdownError <- srv.Shutdown(ctx)
    }()

    app.logger.Info("starting server", "addr", srv.Addr, "env", app.config.env)

    // Calling Shutdown() on our server will cause ListenAndServe() to immediately 
    // return a http.ErrServerClosed error. So if we see this error, it is actually a
    // good thing and an indication that the graceful shutdown has started. So we check 
    // specifically for this, only returning the error if it is NOT http.ErrServerClosed. 
    err := srv.ListenAndServe()
    if !errors.Is(err, http.ErrServerClosed) {
        return err
    }

    // Otherwise, we wait to receive the return value from Shutdown() on the  
    // shutdownError channel. If the return value is an error, we know that there was a
    // problem with the graceful shutdown and we return the error.
    err = <-shutdownError
    if err != nil {
        return err
    }

    // At this point we know that the graceful shutdown completed successfully and we 
    // log a "stopped server" message.
    app.logger.Info("stopped server", "addr", srv.Addr)

    return nil
}

At first glance this code might seem a bit complex, but at a high level what it’s doing can be summarized very simply: when we receive a SIGINT or SIGTERM signal, we instruct our server to stop accepting any new HTTP requests, and give any in-flight requests a ‘grace period’ of 30 seconds to complete before the application is terminated.

It’s important to be aware that the Shutdown() method does not wait for any background tasks to complete, nor does it close hijacked long-lived connections like WebSockets. Instead, you will need to implement your own logic to coordinate a graceful shutdown of these things. We’ll look at some techniques for doing this later in the book.

But that aside, this should now be working nicely in our application.

To help demonstrate the graceful shutdown functionality, you can add a 4 second sleep delay to the healthcheckHandler method, like so:

File: cmd/api/healthcheck.go

package main

import (
    "net/http"
    "time" // New import
)

func (app *application) healthcheckHandler(w http.ResponseWriter, r *http.Request) {
    env := envelope{
        "status": "available",
        "system_info": map[string]string{
            "environment": app.config.env,
            "version":     version,
        },
    }

    // Add a 4 second delay.
    time.Sleep(4 * time.Second)

    err := app.writeJSON(w, http.StatusOK, env, nil)
    if err != nil {
        app.serverErrorResponse(w, r, err)
    }
}

Then start the API, and in another terminal window issue a request to the healthcheck endpoint followed by a SIGTERM signal.

$ curl localhost:4000/v1/healthcheck & pkill -SIGTERM api

In the logs for the server, you should immediately see a "shutting down server" message following the SIGTERM signal, similar to this:

$ go run ./cmd/api
time=2023-09-10T10:59:13.722+02:00 level=INFO msg="database connection pool established"
time=2023-09-10T10:59:13.722+02:00 level=INFO msg="starting server" addr=:4000 env=development
time=2023-09-10T10:59:14.722+02:00 level=INFO msg="shutting down server" signal=terminated

Then after a 4-second delay for the in-flight request to complete, our healthcheckHandler should return the JSON response as normal and you should see that our API has logged a final "stopped server" message before exiting cleanly:

$ go run ./cmd/api
time=2023-09-10T10:59:13.722+02:00 level=INFO msg="database connection pool established"
time=2023-09-10T10:59:13.722+02:00 level=INFO msg="starting server" addr=:4000 env=development
time=2023-09-10T10:59:14.722+02:00 level=INFO msg="shutting down server" signal=terminated
time=2023-09-10T10:59:18.722+02:00 level=INFO msg="stopped server" addr=:4000

Notice the 4 second delay between the "shutting down server" and "stopped server" messages in the timestamps above?

So this is working really well now. Any time that we want to gracefully shut down our application, we can do so by sending a SIGINT (Ctrl+C) or SIGTERM signal. So long as no in-flight requests take more than 30 seconds to complete, our handlers will have time to complete their work and our clients will receive a proper HTTP response. And if we ever want to exit immediately, without a graceful shutdown, we can still do so by sending a SIGQUIT (Ctrl+\) or SIGKILL signal instead.

Lastly, if you’re following along, please revert healthcheckHandler to remove the 4 second sleep. Like so:

File: cmd/api/healthcheck.go

package main

import (
    "net/http"
)

func (app *application) healthcheckHandler(w http.ResponseWriter, r *http.Request) {
    env := envelope{
        "status": "available",
        "system_info": map[string]string{
            "environment": app.config.env,
            "version":     version,
        },
    }

    err := app.writeJSON(w, http.StatusOK, env, nil)
    if err != nil {
        app.serverErrorResponse(w, r, err)
    }
}