Running the API as a background service
Now that we know our API executable works fine on our production droplet, the next step is to configure it to run as a background service, including starting up automatically when the droplet is rebooted.
There are a few different tools we could use to do this, but in this book we will use systemd — a collection of tools for managing services that ships with Ubuntu (and many other Linux distributions).
In order to run our API application as a background service, the first thing we need to do is make a unit file, which informs systemd how and when to run the service.
If you’re following along, head back to a terminal window on your local machine and create a new remote/production/api.service file in your project directory:
$ mkdir remote/production $ touch remote/production/api.service
And then add the following markup:
[Unit] # Description is a human-readable name for the service. Description=Greenlight API service # Wait until PostgreSQL is running and the network is "up" before starting the service. After=postgresql.service After=network-online.target Wants=network-online.target # Configure service start rate limiting. If the service is (re)started more than 5 times # in 600 seconds then don't permit it to start anymore. StartLimitIntervalSec=600 StartLimitBurst=5 [Service] # Execute the API binary as the greenlight user, loading the environment variables from # /etc/environment and using the working directory /home/greenlight. Type=exec User=greenlight Group=greenlight EnvironmentFile=/etc/environment WorkingDirectory=/home/greenlight ExecStart=/home/greenlight/api -port=4000 -db-dsn=${GREENLIGHT_DB_DSN} -env=production # Automatically restart the service after a 5-second wait if it exits with a non-zero # exit code. If it restarts more than 5 times in 600 seconds, then the rate limit we # configured above will be hit and it won't be restarted anymore. Restart=on-failure RestartSec=5 [Install] # Start the service automatically at boot time (the 'multi-user.target' describes a boot # state when the system will accept logins). WantedBy=multi-user.target
Now that we’ve got a unit file set up, the next step is to install this unit file on our droplet and start up the service. Essentially we need to do three things:
- To install the unit file, we need to copy it into the
/etc/systemd/system/folder on our droplet. Because this folder is owned by therootuser on our droplet, we need to break this down into two steps: first we need to copy the unit file into thegreenlightuser’s home directory, and secondly use thesudo mvcommand to move it to its final location. - Then we need to run the
systemctl enable apicommand on our droplet to makesystemdaware of the new unit file and automatically enable the service when the droplet is rebooted. - Finally, we need to run
systemctl restart apito start (or restart) the service.
All three of these tasks will need to be run with sudo.
Let’s update the production/deploy/api makefile rule to automate these tasks for us. Like so:
... # ==================================================================================== # # PRODUCTION # ==================================================================================== # production_host_ip = '161.35.71.158' ... ## production/deploy/api: deploy the api to production .PHONY: production/deploy/api production/deploy/api: rsync -P ./bin/linux_amd64/api greenlight@${production_host_ip}:~ rsync -rP --delete ./migrations greenlight@${production_host_ip}:~ rsync -P ./remote/production/api.service greenlight@${production_host_ip}:~ ssh -t greenlight@${production_host_ip} '\ migrate -path ~/migrations -database $$GREENLIGHT_DB_DSN up \ && sudo mv ~/api.service /etc/systemd/system/ \ && sudo systemctl enable api \ && sudo systemctl restart api \ '
Save the makefile, then go ahead and run the production/deploy/api rule again. You should be prompted to enter the password for the greenlight user when it’s executing, and the rule should complete without any errors. The output you see should look similar to this:
$ make production/deploy/api
rsync -P ./bin/linux_amd64/api greenlight@"161.35.71.158":~
api
7,700,480 100% 2.39GB/s 0:00:00 (xfr#1, to-chk=0/1)
rsync -rP --delete ./migrations greenlight@"161.35.71.158":~
sending incremental file list
migrations/000001_create_movies_table.down.sql
28 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=11/13)
migrations/000001_create_movies_table.up.sql
286 100% 279.30kB/s 0:00:00 (xfr#2, to-chk=10/13)
migrations/000002_add_movies_check_constraints.down.sql
198 100% 193.36kB/s 0:00:00 (xfr#3, to-chk=9/13)
migrations/000002_add_movies_check_constraints.up.sql
289 100% 282.23kB/s 0:00:00 (xfr#4, to-chk=8/13)
migrations/000003_add_movies_indexes.down.sql
78 100% 76.17kB/s 0:00:00 (xfr#5, to-chk=7/13)
migrations/000003_add_movies_indexes.up.sql
170 100% 166.02kB/s 0:00:00 (xfr#6, to-chk=6/13)
migrations/000004_create_users_table.down.sql
27 100% 26.37kB/s 0:00:00 (xfr#7, to-chk=5/13)
migrations/000004_create_users_table.up.sql
294 100% 287.11kB/s 0:00:00 (xfr#8, to-chk=4/13)
migrations/000005_create_tokens_table.down.sql
28 100% 27.34kB/s 0:00:00 (xfr#9, to-chk=3/13)
migrations/000005_create_tokens_table.up.sql
203 100% 198.24kB/s 0:00:00 (xfr#10, to-chk=2/13)
migrations/000006_add_permissions.down.sql
73 100% 71.29kB/s 0:00:00 (xfr#11, to-chk=1/13)
migrations/000006_add_permissions.up.sql
452 100% 441.41kB/s 0:00:00 (xfr#12, to-chk=0/13)
rsync -P ./remote/production/api.service greenlight@"161.35.71.158":~
api.service
1,266 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1)
ssh -t greenlight@"161.35.71.158" '\
migrate -path ~/migrations -database $GREENLIGHT_DB_DSN up \
&& sudo mv ~/api.service /etc/systemd/system/ \
&& sudo systemctl enable api \
&& sudo systemctl restart api \
'
no change
[sudo] password for greenlight:
Created symlink /etc/systemd/system/multi-user.target.wants/api.service → /etc/systemd/system/api.service.
Connection to 161.35.71.158 closed.
Next let’s connect to the droplet and check the status of our new api service using the sudo systemctl status api command:
$ make production/connect
greenlight@greenlight-production:~$ sudo systemctl status api
● api.service - Greenlight API service
Loaded: loaded (/etc/systemd/system/api.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2023-02-23 17:28:18 CET; 1min 7s ago
Main PID: 2717 (api)
Tasks: 5 (limit: 512)
Memory: 2.9M
CPU: 10ms
CGroup: /system.slice/api.service
└─2717 /home/greenlight/api -port=4000 -db-dsn=postgres://greenlight:pa55word1234@localhost/greenlight -env=production
Feb 23 17:28:18 greenlight-production systemd[1]: Starting Greenlight API service...
Feb 23 17:28:18 greenlight-production systemd[1]: Started Greenlight API service.
Feb 23 17:28:18 greenlight-production api[2717]: time=2023-02-23T17:28:18.722+02:00 level=INFO msg="database connection pool established"
Feb 23 17:28:18 greenlight-production api[2717]: time=2023-02-23T17:28:18.722+02:00 level=INFO msg="starting server" ...
Great! This confirms that our api service is running successfully in the background and, in my case, that it has the PID (process ID) 2717.
Out of interest, let’s also quickly list the running processes for our greenlight user:
greenlight@greenlight-production:~$ ps -U greenlight PID TTY TIME CMD 2717 ? 00:00:00 api 2749 ? 00:00:00 systemd 2750 ? 00:00:00 (sd-pam) 2807 ? 00:00:00 sshd 2808 pts/0 00:00:00 bash 2859 pts/0 00:00:00 ps
That tallies — we can see the api process listed here with the PID 2717, confirming that it is indeed our greenlight user who is running the api binary.
Lastly, let’s check from an external perspective and try making a request to our healthcheck endpoint again, either in a browser or with curl. You should get a successful response like so:
$ curl 161.35.71.158:4000/v1/healthcheck
{
"status": "available",
"system_info": {
"environment": "production",
"version": "1c9b6ff48ea800acdf4f5c6f5c3b62b98baf2bd7"
}
}
Restarting after reboot
By running the systemctl enable api command in our makefile after we copied across the systemd unit file, our API service should be started up automatically after the droplet is rebooted.
If you like, you can verify this yourself. While you’re connected to the droplet via SSH, go ahead and reboot it (make sure you are connected to the droplet and don’t accidentally reboot your local machine!).
greenlight@greenlight-production:~$ sudo reboot greenlight@greenlight-production:~$ Connection to 161.35.71.158 closed by remote host. Connection to 161.35.71.158 closed. make: *** [Makefile:91: production/connect] Error 255
Wait a minute for the reboot to complete and for the droplet to come back online. Then you should be able to reconnect and use systemctl status to check that the service is running again. Like so:
$ make production/connect
greenlight@greenlight-production:~$ sudo systemctl status api.service
[sudo] password for greenlight:
● api.service - Greenlight API service
Loaded: loaded (/etc/systemd/system/api.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2023-02-23 17:34:55 CET; 1min 33s ago
Main PID: 895 (api)
Tasks: 5 (limit: 512)
Memory: 8.8M
CPU: 15ms
CGroup: /system.slice/api.service
└─895 /home/greenlight/api -port=4000 -db-dsn=postgres://greenlight:pa55word1234@localhost/greenlight -env=production
Feb 23 17:28:18 greenlight-production systemd[1]: Starting Greenlight API service...
Feb 23 17:28:18 greenlight-production systemd[1]: Started Greenlight API service.
Feb 23 17:28:18 greenlight-production api[2717]: time=2023-02-23T17:28:18.722+02:00 level=INFO msg="database connection pool established"
Feb 23 17:28:18 greenlight-production api[2717]: time=2023-02-23T17:28:18.722+02:00 level=INFO msg="starting server" ...
Disable port 4000
If you’ve been following along with the steps in this section of the book, let’s revert the temporary firewall change that we made earlier and disallow traffic on port 4000 again.
greenlight@greenlight-production:~$ sudo ufw delete allow 4000/tcp Rule deleted Rule deleted (v6) greenlight@greenlight-production:~$ sudo ufw status Status: active To Action From -- ------ ---- 22 ALLOW Anywhere 80/tcp ALLOW Anywhere 443/tcp ALLOW Anywhere 22 (v6) ALLOW Anywhere (v6) 80/tcp (v6) ALLOW Anywhere (v6) 443/tcp (v6) ALLOW Anywhere (v6)
Additional information
Listening on a restricted port
If you’re not planning to run your application behind a reverse proxy, and want to listen for requests directly on port 80 or 443, you’ll need to set up your unit file so that the service has the CAP_NET_BIND_SERVICE capability (which will allow it to bind to a restricted port). For example:
[Unit] Description=Greenlight API service After=postgresql.service After=network-online.target Wants=network-online.target StartLimitIntervalSec=600 StartLimitBurst=5 [Service] Type=exec User=greenlight Group=greenlight CapabilityBoundingSet=CAP_NET_BIND_SERVICE AmbientCapabilities=CAP_NET_BIND_SERVICE EnvironmentFile=/etc/environment WorkingDirectory=/home/greenlight ExecStart=/home/greenlight/api -port=80 -db-dsn=${GREENLIGHT_DB_DSN} -env=production Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target
Viewing logs
It’s possible to view the logs for your background service using the journalctl command, like so:
$ sudo journalctl -u api -- Logs begin at Sun 2021-04-18 14:55:45 EDT, end at Mon 2021-04-19 03:39:55 EDT. -- Apr 19 03:24:35 greenlight-production systemd[1]: Starting Greenlight API service... Apr 19 03:24:35 greenlight-production systemd[1]: Started Greenlight API service. Apr 19 03:24:35 greenlight-production api[6997]: time=2023-02-23T17:28:18.722+02:00 level=INFO msg="database connection pool established" Apr 19 03:24:35 greenlight-production api[6997]: time=2023-02-23T17:28:18.722+02:00 level=INFO msg="starting server" ...
The journalctl command is really powerful and offers a wide variety of parameters that you can use to filter your log messages and customize the formatting. This article provides a great introduction to the details of journalctl and is well worth a read.
Configuring the SMTP provider
Our unit file is currently set up to start our API with the following command:
ExecStart=/home/greenlight/api -port=4000 -db-dsn=${GREENLIGHT_DB_DSN} -env=production
It’s important to remember that apart from the port, db-dsn and env flags that we’re specifying here, our application will still be using the default values for the other settings which are hardcoded into the cmd/api/main.go file — including the SMTP credentials for your Mailtrap inbox. Under normal circumstances, you would want to set your production SMTP credentials as part of this command in the unit file too.
Additional unit file options
Systemd unit files offer a huge range of configuration options, and we’ve only just scratched the surface in this chapter. For more information, the Understanding Systemd Units and Unit Files article is a really good overview, and you can find comprehensive (albeit dense) information in the man pages.
There is also this gist which talks through the various security options that you can use to help harden your service.