Creating and using makefiles
In this first chapter we’re going to look at how to use the GNU make utility and makefiles to help automate common tasks in your project.
The make tool should be pre-installed in most Linux distributions, but if it’s not already on your machine you should be able to install it via your package manager. For example, if your OS supports the apt package manager (like Debian and Ubuntu do) you can install it with:
$ sudo apt install make
It’s also probably already on your machine if you use macOS, but if not you can use brew to install it:
$ brew install make
On Windows machines you can install make using the Chocolatey package manager with the command:
> choco install make
A simple makefile
Now that the make utility is installed on your system, let’s create our first iteration of a makefile. We’ll start simple, and then build things up step-by-step.
A makefile is essentially a text file which contains one or more rules that the make utility can run. Each rule has a target and contains a sequence of commands which are executed when the rule is run. Generally speaking, makefile rules have the following structure:
# comment (optional) target: command command ...
If you’ve been following along, you should already have an empty Makefile in the root of your project directory. Let’s jump in and create a rule which executes the go run ./cmd/api command to run our API application. Like so:
run: go run ./cmd/api
Make sure that the Makefile is saved, and then you can execute a specific rule by running $ make <target> from your terminal.
Let’s go ahead and call make run to start the API:
$ make run go run ./cmd/api time=2023-09-10T10:59:13.722+02:00 level=INFO msg="database connection pool established" time=2023-09-10T10:59:13.722+02:00 level=INFO msg="starting server" addr=:4000 env=development
Great, that’s worked well. When we type make run, the make utility looks for a file called Makefile or makefile in the current directory and then executes the commands associated with the run target.
One thing to point out — by default make echoes commands in the terminal output. We can see that in the code above where the first line of the output is the echoed command go run ./cmd/api. If you want, it’s possible to suppress commands from being echoed by prefixing them with the @ character.
Environment variables
When we execute a make rule, every environment variable that is available to make when it starts is transformed into a make variable with the same name and value. We can then access these variables using the syntax ${VARIABLE_NAME} in our makefile.
To illustrate this, let’s create two additional rules — a psql rule for connecting to our database and an up rule to execute our database migrations. If you’ve been following along, both of these rules will need access to the database DSN value from your GREENLIGHT_DB_DSN environment variable.
Go ahead and update your Makefile to include these two new rules like so:
run: go run ./cmd/api psql: psql ${GREENLIGHT_DB_DSN} up: @echo 'Running up migrations...' migrate -path ./migrations -database ${GREENLIGHT_DB_DSN} up
Notice here how we’ve used the @ character in the up rule to prevent the echo command from being echoed itself when running?
OK, let’s try this out by running make up to execute our database migrations:
$ make up Running up migrations... migrate -path ./migrations -database postgres://greenlight:pa55word@localhost/greenlight up no change
You should see from the output that the value of your GREENLIGHT_DB_DSN environment variable is successfully pulled through and used in the make rule. If you’ve been following along, there shouldn’t be any outstanding migrations to apply, so this should then exit successfully with no further action.
We’re also starting to see the benefits of using a makefile here — being able to type make up is a big improvement on having to remember and use the full command for executing our ‘up’ migrations.
Likewise, if you want, you can also try running make psql to connect to the greenlight database with psql.
Passing arguments
The make utility also allows you to pass named arguments when executing a particular rule. To illustrate this, let’s add a migration rule to our makefile to generate a new pair of migration files. The idea is that when we execute this rule we’ll pass the name of the migration files as an argument, similar to this:
$ make migration name=create_example_table
The syntax to access the value of named arguments is exactly the same as for accessing environment variables. So, in the example above, we could access the migration file name via ${name} in our makefile.
Go ahead and update the makefile to include this new migration rule, like so:
run: go run ./cmd/api psql: psql ${GREENLIGHT_DB_DSN} migration: @echo 'Creating migration files for ${name}...' migrate create -seq -ext=.sql -dir=./migrations ${name} up: @echo 'Running up migrations...' migrate -path ./migrations -database ${GREENLIGHT_DB_DSN} up
And if you execute this new rule with the name=create_example_table argument you should see the following output:
$ make migration name=create_example_table Creating migration files for create_example_table ... migrate create -seq -ext=.sql -dir=./migrations create_example_table /home/alex/Projects/greenlight/migrations/000007_create_example_table.up.sql /home/alex/Projects/greenlight/migrations/000007_create_example_table.down.sql
You’ll now also have two new empty migration files with the name create_example_table in your migrations folder. Like so:
$ ls ./migrations/ 000001_create_movies_table.down.sql 000004_create_users_table.up.sql 000001_create_movies_table.up.sql 000005_create_tokens_table.down.sql 000002_add_movies_check_constraints.down.sql 000005_create_tokens_table.up.sql 000002_add_movies_check_constraints.up.sql 000006_add_permissions.down.sql 000003_add_movies_indexes.down.sql 000006_add_permissions.up.sql 000003_add_movies_indexes.up.sql 000007_create_example_table.down.sql 000004_create_users_table.down.sql 000007_create_example_table.up.sql
If you’re following along, we won’t actually be using these two new migration files, so feel free to delete them:
$ rm migrations/000007*
Namespacing targets
As your makefile continues to grow, you might want to start namespacing your target names to provide some differentiation between rules and help organize the file. For example, in a large makefile rather than having the target name up it would be clearer to give it the name db/migrations/up instead.
I recommend using the / character as a namespace separator, rather than a period, hyphen or the : character. In fact, the : character should be strictly avoided in target names as it can cause problems when using target prerequisites (something that we’ll cover in a moment).
Let’s update our target names to use some sensible namespaces, like so:
run/api: go run ./cmd/api db/psql: psql ${GREENLIGHT_DB_DSN} db/migrations/new: @echo 'Creating migration files for ${name}...' migrate create -seq -ext=.sql -dir=./migrations ${name} db/migrations/up: @echo 'Running up migrations...' migrate -path ./migrations -database ${GREENLIGHT_DB_DSN} up
And you should be able to execute the rules by typing the full target name when running make. For example:
$ make run/api go run ./cmd/api time=2023-09-10T10:59:13.722+02:00 level=INFO msg="database connection pool established" time=2023-09-10T10:59:13.722+02:00 level=INFO msg="starting server" addr=:4000 env=development
A nice feature of using the / character as the namespace separator is that you get tab completion in the terminal when typing target names. For example, if you type make db/migrations/ and then hit tab on your keyboard the remaining targets under the namespace will be listed. Like so:
$ make db/migrations/ new up
Prerequisite targets and asking for confirmation
The general syntax for a makefile rule that I gave at the start of this chapter was a slight simplification, because it’s also possible to specify prerequisite targets.
target: prerequisite-target-1 prerequisite-target-2 ... command command ...
When you specify a prerequisite target for a rule, the corresponding commands for the prerequisite targets will be run before executing the actual target commands.
Let’s leverage this functionality to ask the user for confirmation to continue before executing our db/migrations/up rule.
To do this, we’ll create a new confirm target which asks the user Are you sure? [y/N] and exits with an error if they do not enter y. Then we’ll use this new confirm target as a prerequisite for db/migrations/up.
Go ahead and update your Makefile as follows:
# Create the new confirm target. confirm: @echo -n 'Are you sure? [y/N] ' && read ans && [ $${ans:-N} = y ] run/api: go run ./cmd/api db/psql: psql ${GREENLIGHT_DB_DSN} db/migrations/new: @echo 'Creating migration files for ${name}...' migrate create -seq -ext=.sql -dir=./migrations ${name} # Include it as prerequisite. db/migrations/up: confirm @echo 'Running up migrations...' migrate -path ./migrations -database ${GREENLIGHT_DB_DSN} up
The code in the confirm target is taken from this StackOverflow post. Essentially, what happens here is that we ask the user Are you sure? [y/N] and then read the response. We then use the code [ $${ans:-N} = y ] to evaluate the response — this will return true if the user enters y and false if they enter anything else. If a command in a makefile returns false, then make will stop running the rule and exit with an error message — essentially stopping the rule in its tracks.
Also, importantly, notice that we have set confirm as a prerequisite for the db/migrations/up target?
Let’s try this out and see what happens when we enter y:
$ make db/migrations/up Are you sure? [y/N] y Running up migrations... migrate -path ./migrations -database postgres://greenlight:pa55word@localhost/greenlight up no change
That looks good — the commands in our db/migrations/up rule have been executed as we would expect.
In contrast, let’s try the same thing again but enter any other letter when asked for confirmation. This time, make should exit without executing anything in the db/migrations/up rule. Like so:
$ make db/migrations/up Are you sure? [y/N] n make: *** [Makefile:3: confirm] Error 1
Using a confirm rule as a prerequisite target like this is a really nice reusable pattern. Any time you have a makefile rule which does something destructive or dangerous, you can now just include confirm as a prerequisite target to ask the user for confirmation to continue.
Displaying help information
Another small thing that we can do to make our makefile more user-friendly is to include some comments and help functionality. Specifically, we’ll prefix each rule in our makefile with a comment in the following format:
## <example target call>: <help text>
Then we’ll create a new help rule which parses the makefile itself, extracts the help text from the comments using sed, formats them into a table and then displays them to the user.
If you’re following along, go ahead and update your Makefile so that it looks like this:
## help: print this help message help: @echo 'Usage:' @sed -n 's/^##//p' ${MAKEFILE_LIST} | column -t -s ':' | sed -e 's/^/ /' confirm: @echo -n 'Are you sure? [y/N] ' && read ans && [ $${ans:-N} = y ] ## run/api: run the cmd/api application run/api: go run ./cmd/api ## db/psql: connect to the database using psql db/psql: psql ${GREENLIGHT_DB_DSN} ## db/migrations/new name=$1: create a new database migration db/migrations/new: @echo 'Creating migration files for ${name}...' migrate create -seq -ext=.sql -dir=./migrations ${name} ## db/migrations/up: apply all up database migrations db/migrations/up: confirm @echo 'Running up migrations...' migrate -path ./migrations -database ${GREENLIGHT_DB_DSN} up
And if you now execute the help target, you should get a response which lists all the available targets and the corresponding help text. Similar to this:
$ make help Usage: help print this help message run/api run the cmd/api application db/psql connect to the database using psql db/migrations/new name=$1 create a new database migration db/migrations/up apply all up database migrations
I should also point out that positioning the help rule as the first thing in the Makefile is a deliberate move. If you run make without specifying a target then it will default to executing the first rule in the file.
So this means that if you try to run make without a target you’ll now be presented with the help information, like so:
$ make Usage: help print this help message run/api run the cmd/api application db/psql connect to the database using psql db/migrations/new name=$1 create a new database migration db/migrations/up apply all up database migrations
Phony targets
In this chapter we’ve been using make to execute actions, but another (and arguably, the primary) purpose of make is to help create files on disk where the name of a target is the name of a file being created by the rule.
If you’re using make primarily to execute actions, like we are, then this can cause a problem if there is a file in your project directory with the same path as a target name.
If you want, you can demonstrate this problem by creating a file called ./run/api in the root of your project directory, like so:
$ mkdir run && touch run/api
And then if you execute make run/api, instead of our API application starting up you’ll get the following message:
$ make run/api make: 'run/api' is up to date.
Because we already have a file on disk at ./run/api, the make tool considers this rule to have already been executed and so returns the message that we see above without taking any further action.
To work around this, we can declare our makefile targets to be phony targets:
A phony target is one that is not really the name of a file; rather it is just a name for a rule to be executed.
To declare a target as phony, you can make it prerequisite of the special .PHONY target. The syntax looks like this:
.PHONY: target target: prerequisite-target-1 prerequisite-target-2 ... command command ...
Let’s go ahead and update out Makefile so that all our rules have phony targets, like so:
## help: print this help message .PHONY: help help: @echo 'Usage:' @sed -n 's/^##//p' ${MAKEFILE_LIST} | column -t -s ':' | sed -e 's/^/ /' .PHONY: confirm confirm: @echo -n 'Are you sure? [y/N] ' && read ans && [ $${ans:-N} = y ] ## run/api: run the cmd/api application .PHONY: run/api run/api: go run ./cmd/api ## db/psql: connect to the database using psql .PHONY: db/psql db/psql: psql ${GREENLIGHT_DB_DSN} ## db/migrations/new name=$1: create a new database migration .PHONY: db/migrations/new db/migrations/new: @echo 'Creating migration files for ${name}...' migrate create -seq -ext=.sql -dir=./migrations ${name} ## db/migrations/up: apply all up database migrations .PHONY: db/migrations/up db/migrations/up: confirm @echo 'Running up migrations...' migrate -path ./migrations -database ${GREENLIGHT_DB_DSN} up
If you run make run/api again now, it should now correctly recognize this as a phony target and execute the rule for us:
$ make run/api go run ./cmd/api -db-dsn=postgres://greenlight:pa55word@localhost/greenlight time=2023-09-10T10:59:13.722+02:00 level=INFO msg="database connection pool established" time=2023-09-10T10:59:13.722+02:00 level=INFO msg="starting server" addr=:4000 env=development
You might think that it’s only necessary to declare targets as phony if you have a conflicting file name, but in practice not declaring a target as phony when it actually is can lead to bugs or confusing behavior. For example, imagine if in the future someone unknowingly creates a file called confirm in the root of the project directory. This would mean that our confirm rule is never executed, which in turn would lead to dangerous or destructive rules being executed without confirmation.
To avoid this kind of bug, if you have a makefile rule which carries out an action (rather than creating a file) then it’s best to get into the habit of declaring it as phony.
If you’re following along, you can go ahead and remove the contents of the run directory that we just made. Like so:
$ rm -rf run/
All in all, this is shaping up nicely. Our makefile is starting to contain some helpful functionality, and we’ll continue to add more to it over the next few chapters of this book.
Although this is one of the last things we’re doing in this build, creating a makefile in the root of a project directory is normally one of the very first things I do when starting a project. I find that using a makefile for common tasks helps save both typing and mental overhead during development, and — in the longer term — it acts as a useful entry point and a reminder of how things work when you come back to a project after a long break.