Module proxies and vendoring

Chapter 19.4.

One of the risks of using third-party packages in your Go code is that the package repository may cease to be available. For example, the httprouter package plays a central part in our application, and if the author ever decided to delete it from GitHub it would cause us quite a headache to scramble and replace it with an alternative.

(I’m not suggesting this is likely to happen with httprouter — just using it as an example!)

Fortunately, Go provides two ways in which we can mitigate this risk: module proxies and vendoring.

Module proxies

Go supports module proxies (also known as module mirrors) by default. These are services which mirror source code from the original, authoritative, repositories (such as those hosted on GitHub, GitLab or BitBucket).

Go ahead and run the go env command on your machine to print out the settings for your Go operating environment. Your output should look similar to this:

$ go env
AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/alex/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/alex/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1737714937=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/home/alex/Projects/greenlight/go.mod'
GOMODCACHE='/home/alex/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/alex/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='on'
GOTELEMETRYDIR='/home/alex/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.25.0'
GOWORK=''
PKG_CONFIG='pkg-config'

The important thing to look at here is the GOPROXY setting, which contains a comma-separated list of module mirrors. By default it has the following value:

GOPROXY="https://proxy.golang.org,direct"

The URL https://proxy.golang.org that we see here points to a module mirror maintained by the Go team at Google, containing copies of the source code from tens of thousands of open-source Go packages.

Whenever you fetch a package using the go command — either with go get or one of the go mod * commands — it will first attempt to retrieve the source code from this mirror.

If the mirror already has a stored copy of the source code for the required package and version number, then it will return this code immediately in a zip file. Otherwise, if it’s not already stored, then the mirror will attempt to fetch the code from the authoritative repository, proxy it onwards to you, and store it for future use.

If the mirror can’t fetch the code at all, then it will return an error response and the go tool will fall back to fetching a copy directly from the authoritative repository (thanks to the direct directive in the GOPROXY setting).

Using a module mirror as the first fetch location has a few benefits:

The https://proxy.golang.org module mirror typically stores packages long-term, thereby providing a degree of protection in case the original repository disappears from the internet.
It’s not possible to override or delete a package once it’s stored in the https://proxy.golang.org module mirror. This can help prevent any bugs or problems which might arise if a package author (or an attacker) releases an edited version of the package with the same version number.
Fetching modules from the https://proxy.golang.org mirror can be much faster than getting them from the authoritative repositories.

In most cases, I would generally suggest leaving the GOPROXY setting with its default value.

But if you don’t want to use the module mirror provided by Google, or you’re behind a firewall that blocks it, there are other alternatives like https://goproxy.io and the Microsoft-provided https://athens.azurefd.net that you can try instead. Or you can even host your own module mirror using the open-source Athens and goproxy projects.

For example, if you wanted to switch to using https://goproxy.io as the primary mirror, then use https://proxy.golang.org as a secondary mirror before falling back to a direct fetch, you could update your GOPROXY setting like so:

$ export GOPROXY=https://goproxy.io,https://proxy.golang.org,direct

Or if you want to disable module mirrors altogether, you can simply set the value to direct like so:

$ export GOPROXY=direct

Vendoring

Go’s module mirror functionality is great, and I recommend using it. But it isn’t a silver bullet for all developers and all projects.

For example, perhaps you don’t want to use a module mirror provided by Google or another third-party, but you also don’t want the overhead of hosting your own mirror. Or maybe you need to routinely work in an environment without network access. In those scenarios you probably still want to mitigate the risk of a disappearing dependency, but using a module mirror isn’t possible or appealing.

You should also be aware that the default proxy.golang.org module mirror doesn’t absolutely guarantee that it will store a copy of the module forever. From the FAQs:

proxy.golang.org does not save all modules forever. There are a number of reasons for this, but one reason is if proxy.golang.org is not able to detect a suitable license. In this case, only a temporarily cached copy of the module will be made available, and may become unavailable if it is removed from the original source and becomes outdated.

Additionally, if you need to come back to a ‘cold’ codebase in 5 or 10 years’ time, will the proxy.golang.org module mirror still be available? Hopefully it will — but it’s hard to say for sure.

So, for these reasons, it can still be sensible to vendor your project dependencies using the go mod vendor command. Vendoring dependencies in this way basically stores a complete copy of the source code for third-party packages in a vendor folder in your project.

Let’s demonstrate how to do this. We’ll start by adapting our make tidy rule to also call the go mod verify and go mod vendor commands, like so:

File: Makefile

...

# ==================================================================================== #
# QUALITY CONTROL
# ==================================================================================== #

## tidy: tidy and vendor module dependencies, and format all .go files
.PHONY: tidy
tidy:
	@echo 'Tidying module dependencies...'
	go mod tidy
	@echo 'Verifying and vendoring module dependencies...'
	go mod verify
	go mod vendor
	@echo 'Formatting .go files...'
	go fmt ./...

...

Just to be clear about what’s going on behind-the-scenes here, let’s quickly step through what will happen when we run make tidy:

The go mod tidy command will make sure the go.mod and go.sum files list all the necessary dependencies for our project (and no unnecessary ones).
The go mod verify command will verify that the dependencies stored in your module cache (located on your machine at $GOPATH/pkg/mod) match the checksums in the go.sum file.
The go mod vendor command will then copy the necessary source code from your module cache into a new vendor directory in your project root.

Let’s try this out and run the new tidy rule like so:

$ make tidy
Tidying module dependencies...
go mod tidy
Verifying and vendoring module dependencies...
go mod verify
all modules verified
go mod vendor
Formatting .go files...
go fmt ./...

Once that’s completed, you should see that a new vendor directory has been created containing copies of all the source code along with a modules.txt file. The directory structure in your vendor folder should look similar to this:

$ tree -L 3 ./vendor/
./vendor/
├── github.com
│   ├── BurntSushi
│   │   └── toml
│   ├── julienschmidt
│   │   └── httprouter
│   ├── lib
│   │   └── pq
│   ├── tomasen
│   │   └── realip
│   └── wneessen
│       └── go-mail
├── golang.org
│   └── x
│       ├── crypto
│       ├── exp
│       ├── mod
│       ├── sync
│       ├── text
│       ├── time
│       └── tools
├── honnef.co
│   └── go
│       └── tools
└── modules.txt

Now, when you run a command such as go run, go test or go build, the go tool will recognize the presence of a vendor folder and the dependency code in the vendor folder will be used — rather than the code in the module cache on your local machine.

If you like, go ahead and try running the API application. You should find that everything compiles and continues to work just like before.

$ make run/api 
go run ./cmd/api -db-dsn=postgres://greenlight:pa55word@localhost/greenlight
time=2023-09-10T10:59:13.722+02:00 level=INFO msg="database connection pool established"
time=2023-09-10T10:59:13.722+02:00 level=INFO msg="starting server" addr=:4000 env=development

Because all the dependency source code is now stored in your project repository itself, it’s easy to check it into Git (or an alternative version control system) alongside the rest of your code. This is reassuring because it gives you ownership of all the code used to build and run your applications, kept under version control.

The downside of this, of course, is that it adds size and bloat to your project repository. This is of particular concern in projects that have a lot of dependencies and the repository will be cloned a lot, such as projects where a CI/CD system clones the repository with each new commit.

Let’s also take a quick look in the vendor/modules.txt file that was created. If you’ve been following along it should look similar to this:

File: vendor/modules.txt

# github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c
## explicit; go 1.18
github.com/BurntSushi/toml
github.com/BurntSushi/toml/internal
# github.com/julienschmidt/httprouter v1.3.0
## explicit; go 1.7
github.com/julienschmidt/httprouter
# github.com/lib/pq v1.10.9
## explicit; go 1.13
github.com/lib/pq
github.com/lib/pq/oid
github.com/lib/pq/scram
# github.com/tomasen/realip v0.0.0-20180522021738-f0c99a92ddce
## explicit
github.com/tomasen/realip
# github.com/wneessen/go-mail v0.6.2
## explicit; go 1.16
github.com/wneessen/go-mail
github.com/wneessen/go-mail/internal/pkcs7
github.com/wneessen/go-mail/log
... etc.

This vendor/modules.txt file is essentially a manifest of the vendored packages and their version numbers. When vendoring is being used, the go tool will check that the module version numbers in modules.txt are consistent with the version numbers in the go.mod file. If there’s any inconsistency, then the go tool will report an error.

Lastly, you should avoid making any changes to the code in the vendor directory. Doing so can potentially cause confusion (because the code would no longer be consistent with the original version of the source code) and — besides — running go mod vendor will overwrite any changes you make each time you run it. If you need to change the code for a dependency, it’s much better to fork it and import the forked version instead.

Additional information

Inconsistent vendoring error

Every time you add, remove or upgrade a dependency, make sure to run go mod vendor or the make tidy command immediately afterwards to synchronize the contents of the vendor folder with your go.mod file. If you don’t, you’ll run into an “inconsistent vendoring” error similar to this:

$ make run/api 
go: inconsistent vendoring in /home/alex/Projects/greenlight:
        github.com/foo/bar@v0.1.0: is explicitly 
            required in go.mod, but not marked as explicit in vendor/modules.txt

        To ignore the vendor directory, use -mod=readonly or -mod=mod.
        To sync the vendor directory, run:
                go mod vendor

The ./… pattern

Most of the go tools support the ./... wildcard pattern, like go fmt ./..., go vet ./... and go test ./.... This pattern matches the current directory and all sub-directories, excluding the vendor directory.

Generally speaking, this is useful because it means that we’re not formatting, vetting or testing the code in our vendor directory unnecessarily — and our make audit rule won’t fail due to any problems that might exist within those vendored packages.