Build Once, Promote by Digest: Shipping the Exact Image You Tested

A common release pipeline works like this: tests run on every commit, and when someone pushes the v1.4.2 tag a separate release job checks out the repository again, builds the image from scratch and pushes it to the registry. Same commit, same Dockerfile; nothing looks wrong. Yet the image your tests saw and the image your users pull come from two different builds. This post describes the pattern that collapses them into a single artifact: build once, test the resulting digest, and when the tests pass put the version tag on that same digest.

Same commit, different bytes

A rebuild only gives the same result if every input is fixed. A few of them almost never are:

• A base image tag such as FROM python:3.12-slim moves to a new digest whenever patches land.
• An unpinned apt-get install, an npm install without a lockfile or pip requirements written as ranges pull whatever is newest at build time.
• Timestamps and a build date passed in as a build argument give even a functionally identical image a different digest. You cannot answer "is it the same?" at the byte level.

The upshot: the published image never went through a test. BuildKit can pin timestamps with SOURCE_DATE_EPOCH, but it cannot freeze package repositories. Building once is far cheaper than full reproducibility and gives you the same guarantee directly.

The pattern: candidate, test, promote

Build once per commit and push the image untagged, by digest only. Tests pull that digest; if they pass, the version tag is added to the same digest and nothing is rebuilt.

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --output type=image,name=registry.example.com/app,push-by-digest=true,push=true \
  --metadata-file build.json .

DIGEST=$(jq -r '."containerimage.digest"' build.json)

What moves between jobs is not a tag but the full registry.example.com/app@sha256:... reference, and the test environment is deployed from it. Promotion is one of these commands:

crane tag "registry.example.com/app@${DIGEST}" 1.4.2
oras tag "registry.example.com/app@${DIGEST}" 1.4.2 latest
docker buildx imagetools create -t registry.example.com/app:1.4.2 "registry.example.com/app@${DIGEST}"

test "$(crane digest registry.example.com/app:1.4.2)" = "$DIGEST"

The check on the last line is not ceremony. Called with a single source and no annotations, imagetools create tags an index as is. If the source is a single-platform manifest, though, the default --prefer-index=true wraps it in a new index and the digest changes; adding --annotation changes it too. When moving to another registry, crane copy keeps the digest, while skopeo copy without --all copies only the architecture of the machine it runs on.

Which digest, for multi-arch images

A multi-architecture image is a manifest list (an image index in OCI terms). The index has its own digest and each platform manifest has its own as well; docker buildx imagetools inspect shows both. The digest you promote must be the index digest, otherwise the release ships for a single architecture.

The less obvious part is test coverage. A test that pulls the index on an amd64 machine only runs the amd64 manifest; arm64 has not been tested at all. Either run the tests on every architecture or record which ones were tested. crane digest --platform linux/arm64 gives you the platform digest.

When architectures are built on separate machines, each one pushes by digest and a merge step creates the index:

docker buildx imagetools create -t registry.example.com/app:sha-3f9c2e1 \
  registry.example.com/app@sha256:<amd64-digest> \
  registry.example.com/app@sha256:<arm64-digest>

The candidate index is born here, so the order has to be build, merge, test, promote. With provenance enabled, BuildKit also adds attestation manifests to the index with the platform unknown/unknown, so even a single-platform build is pushed as an index.

Let the image describe itself

A digest does not tell you which commit it came from. OCI's org.opencontainers.image.revision, org.opencontainers.image.version and org.opencontainers.image.source keys exist for that. Labels go into the image config, annotations into the manifest or index.

docker buildx build \
  --label "org.opencontainers.image.revision=$GIT_SHA" \
  --annotation "index,manifest:org.opencontainers.image.revision=$GIT_SHA" \
  --annotation "index,manifest:org.opencontainers.image.version=$VERSION" ...

These are written at build time; an annotation added during promotion creates a new digest. That forces a decision: the version number is fixed before the build. If the binary prints its version in --version output and the number is only chosen after testing, the only way to get it in is to rebuild. I assign the number when the candidate is built; if the candidate fails, the number is burned and the next candidate takes the next one. Version numbers are cheap, a rebuilt release is not.

As a final check, ask the binary: the version and commit printed by docker run --rm "registry.example.com/app@${DIGEST}" --version must match the tag exactly. In Go, pass them explicitly with -ldflags "-X main.version=$VERSION -X main.commit=$GIT_SHA"; in a container build .git is usually outside the build context, so vcs.revision may be missing from go version -m output altogether.

Artifacts that are not images

For desktop installers, CLI binaries and archives, SHA-256 takes the place of the digest. The build job stores the files and a SHA256SUMS list as an artifact, and the test job downloads it. The release job does not build; it downloads the same artifact, verifies it and uploads it:

- uses: actions/download-artifact@v8
  with:
    name: dist
    run-id: ${{ inputs.candidate_run_id }}
    github-token: ${{ secrets.ARTIFACT_READ_TOKEN }}
- run: sha256sum -c SHA256SUMS

Code signing changes the order. An Authenticode signature on Windows or a codesign signature on macOS changes the file's bytes. If you sign in the release step, the tested file and the shipped file differ once again. Sign at the candidate stage and test the signed file.

Signatures and provenance

A cosign signature is bound to a digest. Sign the registry.example.com/app@${DIGEST} reference, not a tag; given a tag, cosign warns, because the tag may point to a different image at that moment. The tag added at promotion does not invalidate the signature. The real payoff is on the deployment side: if the cluster only admits digests signed by a specific identity, an unsigned image rebuilt in some corner of the pipeline cannot run, and "build once" turns from a habit into a mechanical gate.

cosign sign "registry.example.com/app@${DIGEST}"
cosign verify registry.example.com/app:1.4.2 \
  --certificate-identity="$BUILD_IDENTITY" --certificate-oidc-issuer="$OIDC_ISSUER"

Provenance (SLSA) records which source and which build steps produced the image and travels with it. BuildKit generates it with --provenance=mode=max; since mode=max also captures build argument values, do not pass secrets as build arguments. For files, GitHub artifact attestations play the same role and are checked with gh attestation verify.

Pitfalls

Candidate lifetime. Untagged images can get caught by cleanup rules such as tagStatus: untagged in an ECR lifecycle policy. CI artifacts are not permanent either: GitHub Actions keeps them for 90 days by default, and in GitLab artifacts:expire_in decides. If approval waits over a long weekend, the candidate may be gone and the only option left is a rebuild. Give candidates a tag such as sha-<commit> and keep retention longer than your longest approval wait.

The promotion step skips release channels. The old release job did more than push an image; it wrote the update channel file, refreshed the package repository index and published release notes. Replace it with "tag the digest" and those side jobs disappear silently: the tag is in the right place but users never see the new version. List everything the old job did, move it into promotion, and rehearse promotion with dry runs such as imagetools create --dry-run.

Overwriting a tag. If a published 1.4.2 is moved to another digest, machines that cached the image keep running the old one; in Kubernetes the default imagePullPolicy for any tag other than :latest is IfNotPresent, so two different builds run under the same name. Make version tags immutable (IMMUTABLE in ECR, immutability rules in Harbor), use crane copy --no-clobber when copying, and let only floating tags such as latest move.

Testing by tag. If the test job pulls app:candidate while a parallel pipeline pushes a newer candidate to the same tag, the test has validated a different image. Always pass the digest between jobs.

A short checklist

Every commit is built once and pushed by digest. Tests, deployments and signatures refer to the digest. Version and commit are written at build time, promotion adds nothing, and the digest is measured afterwards. For multi-arch images the index is what gets promoted. Version tags are immutable, candidates outlive the approval window, and every side effect of the old release job has moved into the promotion step. Then "did we ship what we tested?" comes down to a single comparison.