Container Build Caching: Layer Order, BuildKit Cache Mounts and Carrying the Cache Across CI Runners

A container image builds in thirty seconds on a developer laptop and takes six minutes in CI, every single time. The cause is usually the same: the laptop has a warm layer cache, while the ephemeral CI runner starts each job with an empty disk. The second most common cause is a Dockerfile written so that the cache breaks in the first few lines. This post covers how the build cache decides what to reuse, how to order a Dockerfile around it, what BuildKit cache mounts add, and how to carry the cache between runners.

How the cache decides

BuildKit computes a key for each instruction, derived from the result of the previous step and the instruction itself. For RUN, if the command string is unchanged the step comes from cache; nobody checks what the command would download today. For COPY and ADD, the content and permissions of the copied files count, modification times do not. Once a step misses the cache, every step after it runs again.

Two practical consequences follow. Things that change often go at the bottom of the file, things that rarely change go at the top. And if RUN apt-get update sits on its own line with an unchanging command string, a package index from months ago stays cached while the install on the next line looks for versions that no longer exist.

Layer order: lockfile first, source code later

This is the change that pays the most. If you copy the whole source tree and then install dependencies, a one-line edit forces every dependency to be downloaded again.

# syntax=docker/dockerfile:1
FROM node:22-bookworm-slim AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci
COPY . .
RUN npm run build

FROM node:22-bookworm-slim
WORKDIR /app
ENV NODE_ENV=production
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci --omit=dev
COPY --from=build /app/dist ./dist
CMD ["node", "dist/server.js"]

As long as the lockfile is unchanged, npm ci is a cache hit. The same applies to requirements.txt or poetry.lock in Python and go.mod plus go.sum in Go. The multi-stage layout does a second job here: compilers and dev dependencies never reach the final image, and the final stage's layers have fewer reasons to change.

.dockerignore is part of the same design. If COPY . . also picks up .git, a local node_modules, test output or editor files, every commit changes the content and the cache breaks. List everything that has no business in the build context.

Cache mounts: where layer caching falls short

Layer caching is all or nothing. Add one package to the lockfile and npm ci runs from scratch, downloading hundreds of packages again. RUN --mount=type=cache fills that gap: the target directory is mounted as a cache that persists across builds and never ends up in the image. The package manager picks up whatever it already downloaded.

Typical targets are /root/.npm for npm, /root/.cache/pip for pip, /root/.cache/go-build and /go/pkg/mod for Go, and /root/.m2 for Maven. If you build as a non-root user, the path follows that user's home directory. A wrong path produces no error; it just saves nothing.

Official Debian-based images have an apt trap. They ship a config file that deletes downloaded packages after every install, so a mounted cache directory stays empty. The pattern from Docker's documentation removes it:

RUN rm -f /etc/apt/apt.conf.d/docker-clean; \
    echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && apt-get install -y --no-install-recommends ca-certificates curl

sharing=locked stops two concurrent builds from fighting over apt's lock. Keeping update and install in one RUN also fixes the stale index problem described above.

Keep secrets out of the cache and the image

Passing a private registry token through ARG is a common mistake. The value can show up in image history, and every time it changes the cache breaks. Use a secret mount instead:

RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci

docker buildx build --secret id=npmrc,src=$HOME/.npmrc -t app:dev .

The file is mounted only for the duration of that RUN, is never written to a layer, and is not part of the cache key. Rotating the token does not invalidate the build.

The other ARG trap is version metadata. Declare the commit SHA as an ARG near the top and use it in a label or environment variable, and since it changes on every commit, nothing after its first use comes from cache. Put such values at the very end of the final stage, only where they are actually needed.

Carrying the cache in CI

An ephemeral runner has no local cache, so you have to export the cache after a build and import it in the next. BuildKit offers several backends; three approaches matter.

Inline cache (type=inline) embeds cache metadata in the image itself. It is the easiest to set up, but it only covers the final image's layers. In a multi-stage build the build stage's layers are lost, and that is usually where the time goes.

Registry cache (type=registry) writes the cache to the image registry under its own tag. With mode=max it includes layers from intermediate stages. The default, mode=min, writes only the final image's layers.

The CI provider's cache (such as type=gha on GitHub Actions) needs no extra registry, but it is subject to the provider's quota and eviction rules and does not travel to another CI system.

Our pick is the registry cache with mode=max. You already push images to a registry, the cache sits in the same place behind the same credentials, it is independent of the CI vendor, and developers can read the same cache locally:

docker buildx create --name ci --driver docker-container --use
docker buildx build \
  --cache-from type=registry,ref=registry.example.com/app:cache-main \
  --cache-from type=registry,ref=registry.example.com/app:cache-$BRANCH \
  --cache-to type=registry,ref=registry.example.com/app:cache-$BRANCH,mode=max \
  -t registry.example.com/app:$GIT_SHA --push .

The first line matters: the default docker driver cannot export most external cache types, so you create a builder with the docker-container driver. Multiple --cache-from entries are allowed, so a branch that has not produced its own cache yet falls back to main's. A separate write tag per branch keeps parallel branches from constantly overwriting each other.

Pitfalls

Cache mounts are not exported. --cache-to carries the layer cache, not the contents of --mount=type=cache directories. On an ephemeral runner those directories are empty on every job. In CI the real gains come from layer order and the registry cache; cache mounts shine on persistent builders and developer machines.

The base image does not refresh. Once FROM node:22 has been pulled locally, BuildKit may not check for a newer one. Use --pull on regular builds to get security patches, or pin the base image by digest and let a dependency update bot bump it.

The cache carries stale packages. A cached apt-get install step contains packages from weeks ago. A scheduled weekly build with --no-cache takes care of that.

Registry cache grows. mode=max writes every intermediate stage. Without a retention rule that deletes branch cache tags when the branch goes away, registry storage quietly balloons.

Measure your hit rate. Count the steps marked CACHED in the build output. If the dependency install step reruns on every build, the reason is almost always a COPY before it or an ARG whose value changes.

When this is overkill

If the build already finishes in under a minute and runs a few times a day, a registry cache costs more upkeep than it saves; fixing layer order and writing a .dockerignore is enough. If you use a single persistent builder, the local cache is already there and exporting it adds nothing. If a compliance requirement says release images must be built clean, restrict caching to development and verification builds, build releases with --no-cache, and promote the resulting image by digest.