Table of contents · 10 sections

Empty working directory — this is a from-scratch writing task, so here's the tutorial.

---

Docker Layer Caching Strategies That Actually Work

Most Dockerfile "optimization" advice stops at "put COPY package.json before COPY .". That's a real technique, but it's one rule out of a system — and it's why so many teams have a beautifully ordered Dockerfile that still rebuilds from scratch on every CI run.

This tutorial builds the whole model: how BuildKit computes cache keys, what actually invalidates them, how cache mounts differ from layer cache (and why that difference wrecks CI), how to persist cache across ephemeral runners, and how to debug a miss instead of guessing at it.

Everything here assumes BuildKit, which has been the default builder since Docker Engine 23. Start every Dockerfile with a syntax directive so you get current frontend features regardless of the engine version:

# syntax=docker/dockerfile:1

That line pulls the latest stable Dockerfile frontend at build time. Without it you're pinned to whatever your engine shipped with, and half the techniques below (cache mounts, COPY --link, --parents) may not parse.

---

Part 1: The mental model

An image is a stack of diffs

A Docker image is an ordered list of content-addressed filesystem layers plus a config blob. Each layer is a tarball of changes relative to the layer beneath it. RUN, COPY, and ADD produce filesystem layers. ENV, WORKDIR, USER, LABEL, ARG, EXPOSE, ENTRYPOINT and friends produce no filesystem diff — but they do mutate the image config, which matters for caching in a way people consistently miss (more on this shortly).

The cache key

For each build step, BuildKit computes a cache key. If a previously computed result exists for that key, the step is reused and marked CACHED. The key is derived differently per instruction:

Instruction	What goes into the cache key
FROM	Resolved image digest (not the tag string — the digest the tag resolves to)
RUN	Parent cache key + the literal command string (post-variable-substitution) + mount definitions
COPY / ADD (local)	Parent cache key + content checksum of the source files, including path and file mode
ADD (remote URL)	Parent cache key + URL, plus revalidated remote metadata — unless --checksum is given, in which case the checksum
ENV, WORKDIR, LABEL, USER, …	Parent cache key + the instruction text

Two properties fall out of this, and they explain roughly 90% of cache behavior:

1. Keys chain. Every step's key includes its parent's key. A miss at step 4 guarantees a miss at steps 5 through 30, even if those steps are byte-identical to last build. This is the cascade rule, and it's the single most important thing to internalize. Cache invalidation is not per-instruction; it's a suffix of your Dockerfile.

2. RUN does not look inside the container. BuildKit has no idea what RUN apt-get install -y curl will actually do. It hashes the string apt-get install -y curl. If upstream publishes a new curl tomorrow, the string is unchanged, so you get a cache hit and yesterday's curl. The cache is a memoization keyed on instructions, not on outcomes. Every stale-package bug you've hit is this.

COPY is the opposite: it's keyed on content. Touch a file without changing its bytes and you still get a hit. Change a byte and back again, and the key returns to its original value — the cache is content-addressed, not sequential.

Watch it happen

# syntax=docker/dockerfile:1
FROM alpine:3.20
RUN echo "step one" && sleep 2
COPY app.txt /app.txt
RUN echo "step three" && sleep 2

echo hello > app.txt
docker build -t cachedemo .
docker build -t cachedemo .           # everything CACHED, near-instant

touch app.txt
docker build -t cachedemo .           # still fully CACHED — mtime is not content

echo goodbye > app.txt
docker build --progress=plain -t cachedemo .

That last build shows the cascade: step one is CACHED, the COPY misses, and step three re-runs despite being identical text.

---

Part 2: Ordering — the volatility gradient

The governing principle: order instructions from least-frequently-changing to most-frequently-changing.

Rank the inputs to your build by how often they change per unit of developer time:

1. Base image — weeks
2. OS packages — weeks
3. Language runtime / toolchain setup — weeks
4. Dependency manifests (package-lock.json, go.sum, poetry.lock, Cargo.lock) — days
5. Application source — minutes
6. Build metadata (git SHA, build date, version labels) — every single build

Anything out of order pulls everything below it into the invalidation cascade. Build metadata at the top of your Dockerfile means you have no cache at all.

The dependency-manifest split

This is the canonical pattern, and it exists to exploit the fact that manifests change far less often than source:

# syntax=docker/dockerfile:1
FROM node:22-bookworm-slim
WORKDIR /app

# Layer keyed only on the lockfile — survives every source edit
COPY package.json package-lock.json ./
RUN npm ci

# Layer keyed on source — misses constantly, but it's cheap
COPY . .
RUN npm run build

Edit a React component, and npm ci stays cached. Add a dependency, and it correctly re-runs. That's the whole trick: you're separating the slow, rarely-invalidated step from the fast, always-invalidated one.

Equivalents across ecosystems:

# Python (pip)
COPY requirements.txt .
RUN pip install -r requirements.txt

# Python (uv) — install deps without the project itself, then the project
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-install-project
COPY . .
RUN uv sync --frozen

# Go
COPY go.mod go.sum ./
RUN go mod download

# Rust — see the cargo-chef section; Cargo needs a fake source tree
# Java (Maven)
COPY pom.xml .
RUN mvn -B dependency:go-offline

# Ruby
COPY Gemfile Gemfile.lock ./
RUN bundle install

Be precise with the glob. COPY package*.json ./ looks tidy but silently matches package-lock.json, packages.json, and anything else fitting the pattern — and in a monorepo it won't match nested workspace manifests at all, so npm ci will fail or under-install. List files explicitly.

The monorepo manifest problem

Workspaces break the simple pattern: you need package.json from twelve nested directories, preserving structure, without dragging in source. The labs frontend solves this with --parents:

# syntax=docker/dockerfile:1-labs
FROM node:22-bookworm-slim
WORKDIR /app
COPY --parents package.json package-lock.json ./
COPY --parents packages/*/package.json ./
RUN npm ci
COPY . .

--parents preserves the directory structure of matched paths rather than flattening them into the destination. Without it, your options are one COPY per workspace (brittle, but explicit and dependency-free) or a pre-build step that extracts manifests into a staging directory.

.dockerignore is a caching tool, not just a hygiene tool

COPY . . hashes the entire build context. If .git/ is in scope, every commit changes the context hash, so COPY . . misses on every build even when no source file changed. If node_modules/ is in scope, you're hashing and transferring hundreds of megabytes on every build, and your host's node_modules may shadow the container's.

# .dockerignore
.git
.gitignore
node_modules
**/node_modules
dist
build
target
.venv
__pycache__
*.pyc
.pytest_cache
coverage
.env*
*.log
Dockerfile*
docker-compose*.yml
.github
README.md

Two things to know:

• BuildKit transfers the context incrementally and caches it on the builder, so the transfer cost is smaller than it used to be — but the hashing cost and the invalidation cost are entirely real.
• Excluding .env* is a security control as much as a caching one. Secrets in the context are secrets in your layers.

Check your context size directly:

docker build --no-cache --progress=plain . 2>&1 | grep "transferring context"

If that's over a few megabytes for a typical application, your .dockerignore is incomplete.

Metadata goes last

# ✗ Invalidates the entire rest of the build on every commit
ARG GIT_SHA
LABEL org.opencontainers.image.revision=$GIT_SHA
RUN npm ci
COPY . .

# ✓ Invalidates nothing but a metadata-only layer
RUN npm ci
COPY . .
ARG GIT_SHA
LABEL org.opencontainers.image.revision=$GIT_SHA

ARG has a useful subtlety: declaring it doesn't invalidate anything by itself. BuildKit invalidates only the steps that reference it, because substitution happens into the command string. So ARG GIT_SHA declared at the top is harmless; the LABEL that consumes it is what cuts the chain. ENV is stricter — it's a config mutation with no usage tracking, so changing any ENV value invalidates everything after it unconditionally.

---

Part 3: Instruction-level gotchas

Fuse apt-get update with apt-get install

# ✗ Broken across time
RUN apt-get update
RUN apt-get install -y curl ca-certificates

# ✓
RUN apt-get update \
 && apt-get install -y --no-install-recommends curl ca-certificates \
 && rm -rf /var/lib/apt/lists/*

In the broken version, apt-get update gets cached indefinitely. Weeks later you add a package to the install line; the install line's string changed so it re-runs, but it re-runs against a package index cached from weeks ago. You get "404 Not Found" on package fetches, or worse, silently stale versions. This failure mode is called cache busting in the Docker docs and it is genuinely one of the most common production Docker bugs.

Fusing them means the update always runs alongside the install. rm -rf /var/lib/apt/lists/* in the same RUN keeps the index out of the layer — deleting it in a later RUN would not, since layers only add.

--no-install-recommends is a size optimization, not a cache one, but it belongs in the same muscle memory.

Pin base images by digest when you need determinism

FROM node:22-bookworm-slim@sha256:2ba18a1b1a...

FROM node:22-bookworm-slim resolves the tag to a digest at build time. When upstream repushes that tag, the digest changes, the FROM cache key changes, and your entire build invalidates — surprising if you didn't know upstream published. Digest pinning makes that explicit: the build is fully reproducible and cache-stable until you bump the pin, which you should automate with Renovate or Dependabot rather than deferring to whenever upstream feels like it.

The tradeoff is real: pinned digests mean you don't get security patches until something bumps the pin. Pin plus automated bumps; don't pin and forget.

ADD from a URL needs a checksum

# ✗ Cache behavior depends on remote server metadata
ADD https://example.com/tool.tar.gz /tmp/

# ✓ Deterministic key, and it's an integrity check
ADD --checksum=sha256:9f3c... https://example.com/tool.tar.gz /tmp/

Without --checksum, BuildKit revalidates the remote resource on each build and keys off what the server reports. That's a network round-trip on every build and a nondeterministic cache key controlled by a third party. With --checksum, the key is the checksum you wrote, so the step is cached until you change it — and a tampered artifact fails the build.

COPY --link: breaking the dependency chain

This is the most underused caching feature in Dockerfile, and it's the one exception to the cascade rule.

COPY --link ./src /app/src

Normally a COPY layer is computed on top of the parent's filesystem, so its cache key includes the parent's key. --link creates the layer independently — the files are snapshotted into a standalone layer that gets stitched on at assembly time. Its cache key does not include the parent chain.

The consequence: changing your base image no longer invalidates your COPY layers. Bump FROM node:22.1 to node:22.2 and your source layers stay cached and don't need re-uploading. In a multi-stage build where the final stage is a handful of COPY --from instructions, this makes base image bumps nearly free.

Caveats worth respecting:

• The destination's prior contents aren't visible to the copy. If you're copying over existing files and relying on merge semantics, behavior differs.
• --chown with names (--chown=node:node) needs /etc/passwd from the parent image, which isn't part of an independent layer. Use numeric IDs (--chown=1000:1000) with --link.
• Older engines that don't understand the link semantics will fall back or error; the # syntax directive plus a modern engine avoids this.

Use it aggressively in final stages, carefully where you're overlaying onto existing directory trees.

RUN --mount=type=bind instead of COPY for build-only files

If a file is needed during a RUN but shouldn't live in the image, bind-mount it:

RUN --mount=type=bind,source=package.json,target=package.json \
    --mount=type=bind,source=package-lock.json,target=package-lock.json \
    npm ci

No layer is produced for the mounted files. The cache key still incorporates their content, so invalidation is correct — you just don't pay a layer for it.

RUN --mount=type=secret for private registries

RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci

docker build --secret id=npmrc,src=$HOME/.npmrc .

The alternative — ARG NPM_TOKEN — bakes the token into the build history and makes every dependency layer's cache key a function of a rotating credential. Secrets mounts fix both problems: nothing lands in a layer, and the secret's value is excluded from the cache key, so rotating a token doesn't trigger a full rebuild.

There's a matching --mount=type=ssh for private Git dependencies.

---

Part 4: Cache mounts — a completely different cache

Layer caching memoizes whole steps. Cache mounts persist a directory across builds. They solve a different problem, and conflating them is the source of a lot of confusion.

RUN --mount=type=cache,target=/root/.npm \
    npm ci

/root/.npm is a persistent directory on the builder, mounted into the container for the duration of that RUN, and not committed to the resulting layer. So when the step does miss — you added a dependency — npm still finds most packages already downloaded and only fetches the delta.

The two caches compose: layer caching gives you "skip the step entirely," cache mounts give you "the step, when it must run, runs fast."

The rule that makes cache mounts safe

Download into the mount; install into the layer. Anything written to the mount path is invisible to the final image. This is correct for package manager download caches, compiler object caches, and build scratch directories. It is a silent disaster if you point a cache mount at a directory you expect to ship — node_modules, /app/target, site-packages. You'll get a build that works locally (the mount happens to be populated) and an image that's missing its dependencies.

If you must build into a cached directory, copy the artifact out in the same RUN:

RUN --mount=type=cache,target=/app/target \
    --mount=type=cache,target=/usr/local/cargo/registry \
    cargo build --release && cp target/release/myapp /usr/local/bin/myapp

The cp is not optional. Split it into a second RUN and /app/target is empty again.

Sharing modes

Mode	Behavior	Use for
shared (default)	Concurrent builds mount the same directory simultaneously	Content-addressed caches: npm, Go modules, pip wheels
locked	Second build waits for the first to release	Caches with lockfiles or non-atomic writes: apt, Cargo registry
private	Each concurrent build gets its own copy	Caches that corrupt under any sharing

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt/lists,sharing=locked \
    apt-get update && apt-get install -y --no-install-recommends curl

Note that this apt version doesn't rm -rf /var/lib/apt/lists/* — the lists live in a mount, so they're already absent from the layer. One extra step is required on Debian/Ubuntu base images, which ship a hook that deletes downloaded .deb files immediately:

RUN rm -f /etc/apt/apt.conf.d/docker-clean \
 && echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache

Without that, your apt cache mount stays empty and you've achieved nothing.

Use id= to scope a mount when a single target path serves multiple purposes across stages:

RUN --mount=type=cache,id=go-build-amd64,target=/root/.cache/go-build ...

This matters for multi-platform builds, where two architectures writing to one Go build cache will thrash each other.

Per-ecosystem recipes

# npm
RUN --mount=type=cache,target=/root/.npm npm ci

# pnpm — set store dir explicitly
ENV PNPM_HOME=/pnpm
RUN --mount=type=cache,target=/pnpm/store \
    pnpm install --frozen-lockfile

# yarn (berry)
RUN --mount=type=cache,target=/root/.yarn/berry/cache \
    yarn install --immutable

# pip
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

# uv
ENV UV_LINK_MODE=copy
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --frozen --no-install-project

# Go — module cache and build cache are separate and both matter
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    go build -o /out/server ./cmd/server

# Maven
RUN --mount=type=cache,target=/root/.m2 mvn -B package -DskipTests

# Gradle
RUN --mount=type=cache,target=/root/.gradle gradle build --no-daemon

# apt (with the docker-clean fix above)
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt/lists,sharing=locked \
    apt-get update && apt-get install -y --no-install-recommends build-essential

UV_LINK_MODE=copy deserves a note: uv hardlinks from its cache by default, which fails across the mount boundary and emits warnings. Copy mode is the correct setting inside containers.

The thing that will bite you in CI

Cache mounts live on the builder. They are not part of the image, and --cache-to does not export them.

On your laptop, where the same BuildKit daemon persists across builds, cache mounts are transformative. On a fresh GitHub Actions runner with a brand-new builder, they're empty on every run and contribute exactly nothing. Teams routinely add cache mounts, see a 5× local speedup, ship it, and are baffled when CI doesn't move.

Options, in order of preference:

1. Use a persistent builder. A self-hosted runner with a long-lived docker-container builder, a remote BuildKit instance, or Docker Build Cloud. The cache mounts just work, because the daemon is the same one as last build.
2. Lean harder on layer cache. Registry-backed layer cache does survive ephemeral runners. Structure the Dockerfile so the expensive steps hit layer cache and you never need the mount.
3. Save/restore the mount contents. The reproducible-containers/buildkit-cache-dance action extracts cache mount directories into the Actions cache and restores them. It works, but it's fiddly, adds minutes of its own, and I'd reach for it only after (1) and (2) are exhausted.

---

Part 5: Multi-stage builds and cache

Multi-stage builds are usually pitched as a size optimization. They're also a cache isolation mechanism.

# syntax=docker/dockerfile:1

FROM node:22-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

FROM node:22-bookworm-slim AS build
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build

FROM node:22-bookworm-slim AS test
WORKDIR /app
COPY --from=build /app ./
RUN npm test

FROM gcr.io/distroless/nodejs22-debian12 AS runtime
WORKDIR /app
COPY --link --from=build /app/dist ./dist
COPY --link --from=deps /app/node_modules ./node_modules
CMD ["dist/server.js"]

Three properties worth naming:

Stages build in parallel. BuildKit constructs a DAG and executes independent stages concurrently. Two stages that don't depend on each other genuinely run at the same time — this is a real speedup that the legacy builder never had.

Unused stages are skipped entirely. --target build doesn't execute test. BuildKit prunes anything the target doesn't depend on.

Invalidation is scoped by the DAG, not by file order. The test stage can be invalidated without touching runtime's inputs. Changing a test file rebuilds test; runtime only depends on build's output and deps, so it may still be fully cached.

That last point interacts with --cache-to mode= in a way covered in Part 6: intermediate stages are only in your remote cache if you asked for mode=max.

The "dependency plan" pattern for compiled languages

Rust's Cargo.toml doesn't cleanly separate dependency resolution from compilation — cargo build wants a real source tree. cargo-chef generates a synthetic one:

# syntax=docker/dockerfile:1
FROM rust:1-slim AS chef
RUN cargo install cargo-chef
WORKDIR /app

FROM chef AS planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
# Cached unless the dependency graph itself changed
RUN --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/app/target,sharing=locked \
    cargo chef cook --release --recipe-path recipe.json
COPY . .
RUN --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/app/target,sharing=locked \
    cargo build --release && cp target/release/myapp /usr/local/bin/myapp

FROM debian:bookworm-slim AS runtime
COPY --link --from=builder /usr/local/bin/myapp /usr/local/bin/myapp
CMD ["myapp"]

The planner stage reduces the whole source tree to a recipe.json containing only the dependency graph. Editing application source produces an identical recipe, so cargo chef cook — which compiles every dependency — stays cached. This generalizes: whenever a language's dependency step can't be isolated by copying a manifest, look for a way to project the source tree down to just its dependency signature.

---

Part 6: Making cache survive CI

Why your CI has no cache by default

Every CI job gets a clean machine with an empty BuildKit state. Layer cache is local by default, so there is nothing to hit. Fixing this requires exporting cache to somewhere durable and importing it next run — and requires the right builder driver.

Driver matters

Driver	Cache export support
docker (default, the built-in daemon builder)	Inline cache only. Cannot do --cache-to type=registry/gha
docker-container	Full support for all cache backends
kubernetes	Full support
remote	Full support; connects to an existing BuildKit instance
cloud	Docker Build Cloud; persistent shared cache including cache mounts

If --cache-to type=registry errors with "cache export feature is currently not supported for docker driver", this is why. docker/setup-buildx-action creates a docker-container builder, which is why every working GHA example starts with it.

docker buildx create --name ci --driver docker-container --use

Cache backends

Backend	Where it lives	mode=max?	Best for
inline	Embedded in the pushed image's metadata	No	Simplest setup; single-stage builds
registry	A separate image ref in your registry	Yes	The default good answer for most CI
gha	GitHub Actions cache service	Yes	GitHub Actions, when you don't want cache in your registry
s3 / azblob	Object storage	Yes	Large orgs, non-GitHub CI, fine-grained retention control
local	A directory on the runner	Yes	Composing with a generic CI cache action

mode=min vs mode=max is the decision people get wrong. min exports only the layers present in the final image. max exports layers from every stage. In a multi-stage build, min means your compiler stage, dependency stage, and test stage are all absent from the cache — so CI rebuilds them every time while reporting that "caching is enabled." Use mode=max unless cache storage cost is a genuine constraint. inline only does min, which is why it's a poor fit for multi-stage builds.

A working GitHub Actions setup

name: build
on: [push, pull_request]

jobs:
  docker:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4

      - uses: docker/setup-buildx-action@v3

      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - uses: docker/build-push-action@v6
        with:
          context: .
          push: ${{ github.event_name != 'pull_request' }}
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: |
            type=registry,ref=ghcr.io/${{ github.repository }}:buildcache
          cache-to: |
            type=registry,ref=ghcr.io/${{ github.repository }}:buildcache,mode=max

Some registries — notably ECR — need the cache manifest expressed as a normal image manifest:

cache-to: type=registry,ref=...,mode=max,image-manifest=true,oci-mediatypes=true

For the GHA backend instead:

cache-from: type=gha
cache-to: type=gha,mode=max

Keep buildx current for type=gha — GitHub retired the v1 Actions cache service, and older buildx versions talk only to the retired API. If GHA cache silently stopped working for you, check your buildx version first.

Branch-scoped cache with a fallback

A single mutable buildcache tag means concurrent branch builds overwrite each other's cache. Scope per branch, and fall back to the default branch:

env:
  CACHE_REF: ghcr.io/${{ github.repository }}:cache
  BRANCH: ${{ github.head_ref || github.ref_name }}

# ...
  cache-from: |
    type=registry,ref=${{ env.CACHE_REF }}-${{ env.BRANCH }}
    type=registry,ref=${{ env.CACHE_REF }}-main
  cache-to: |
    type=registry,ref=${{ env.CACHE_REF }}-${{ env.BRANCH }},mode=max

Multiple cache-from entries are tried in order — your branch's cache first, then main's as a warm baseline for new branches. This one change is often the difference between "first push to a PR branch takes 12 minutes" and "takes 90 seconds."

For the gha backend, the analogous knob is scope. Note that GitHub's cache isolation rules apply: a PR branch can read the base branch's cache but not vice versa, and caches are scoped so sibling branches can't see each other.

Multi-platform builds need separate cache scopes

Cache for linux/amd64 and linux/arm64 are distinct entries. Sharing one scope makes them evict each other:

cache-from: type=gha,scope=${{ matrix.platform }}
cache-to: type=gha,mode=max,scope=${{ matrix.platform }}

Building both platforms in one job under QEMU is also worth reconsidering — emulated arm64 compilation is often 5–10× slower than native, and native runners per platform plus a manifest-merge step will usually beat any caching improvement you can make.

Storage limits and eviction

• GitHub Actions cache: 10 GB per repository, LRU eviction, plus eviction of entries unused for 7 days. mode=max on a large image can consume that alone, which then evicts your test caches. Watch total usage.
• Registry cache: unbounded but you pay for it. Set registry retention/lifecycle rules on the cache tag — cache manifests accumulate.
• Local builder: BuildKit garbage-collects on its own schedule, configurable in buildkitd.toml.

docker buildx du --verbose                     # what's taking space
docker buildx prune --filter until=168h        # drop anything unused for a week
docker buildx prune -a                         # nuke it

Newer buildx replaces --keep-storage with --reserved-space, --max-used-space, and --min-free-space; check docker buildx prune --help for what your version accepts.

---

Part 7: Measuring and debugging cache misses

Do not optimize by intuition. BuildKit tells you exactly what it did.

Read the build output

docker buildx build --progress=plain -t myapp . 2>&1 | tee build.log

#8 [deps 3/3] RUN npm ci
#8 CACHED

#9 [build 2/4] COPY . .
#9 DONE 0.4s

#10 [build 3/4] RUN npm run build
#10 62.3s

Scan for the first non-CACHED step. Everything after it is collateral damage from the cascade — there's exactly one bug to find, and it's at that boundary. Then ask: is this step supposed to have missed?

Bisect with --no-cache-filter

To confirm a hypothesis about which stage is slow, force a specific stage to rebuild while leaving the rest cached:

docker buildx build --no-cache-filter=deps -t myapp .

This is the right tool for measuring the true cost of a step, and for verifying that a stage genuinely rebuilds when it should (rather than being cached against something stale).

The systematic debug checklist

When a step misses and you think it shouldn't:

1. Is it the first miss, or downstream of one? Fix the first one; the rest usually resolve themselves.
2. Is it FROM? The tag was repushed upstream. Pin by digest.
3. Is it a COPY? Something in the source set changed content, path, or mode. File permissions are part of the checksum — a git clone on Windows or a chmod in a prior CI step will change modes and miss even with identical bytes.
4. Is .git or a build output directory in the context? Check transferring context size and your .dockerignore.
5. Is a rotating value flowing into an early instruction? A GIT_SHA build arg, a timestamp, an ENV set from CI metadata.
6. In CI: is the driver docker-container? The default docker driver can't import registry cache at all.
7. In CI: is mode=max set? Without it, multi-stage intermediates were never exported to import from.
8. In CI: does the imported cache ref actually exist? docker buildx imagetools inspect ghcr.io/org/repo:cache. A typo in the ref fails silently — BuildKit treats a missing cache source as "no cache," not as an error.
9. Was the cache evicted? GHA's 7-day/10 GB limits, or a registry lifecycle rule.
10. Are you relying on cache mounts in ephemeral CI? They're empty every run. See Part 4.

A note on image digest churn

If your image digest changes on every build even when nothing changed, timestamps are the usual cause — BuildKit stamps layer creation times. For bit-reproducible images:

SOURCE_DATE_EPOCH=$(git log -1 --format=%ct) \
  docker buildx build --output type=image,name=myapp,rewrite-timestamp=true .

This doesn't affect cache hit rate directly, but it does mean identical inputs produce an identical digest, which makes "did anything actually change?" answerable and lets registries deduplicate.

---

Part 8: Reference Dockerfiles

Node (multi-stage, distroless runtime)

# syntax=docker/dockerfile:1

FROM node:22-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci

FROM node:22-bookworm-slim AS build
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build
RUN --mount=type=cache,target=/root/.npm \
    npm ci --omit=dev

FROM gcr.io/distroless/nodejs22-debian12 AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --link --from=build /app/node_modules ./node_modules
COPY --link --from=build /app/dist ./dist
USER 1000:1000
CMD ["dist/server.js"]

Python (uv)

# syntax=docker/dockerfile:1

FROM python:3.13-slim-bookworm AS build
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
WORKDIR /app
ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy

RUN --mount=type=cache,target=/root/.cache/uv \
    --mount=type=bind,source=uv.lock,target=uv.lock \
    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
    uv sync --frozen --no-install-project --no-dev

COPY . .
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --frozen --no-dev

FROM python:3.13-slim-bookworm AS runtime
WORKDIR /app
COPY --link --from=build /app /app
ENV PATH="/app/.venv/bin:$PATH"
USER 1000:1000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

The bind mounts for uv.lock and pyproject.toml mean those files never produce a layer in the build stage, while still keying the cache correctly.

Go (static binary, scratch runtime)

# syntax=docker/dockerfile:1

FROM golang:1.23-bookworm AS build
WORKDIR /src
ARG TARGETOS TARGETARCH

RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=bind,source=go.mod,target=go.mod \
    --mount=type=bind,source=go.sum,target=go.sum \
    go mod download

RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,id=gobuild-${TARGETARCH},target=/root/.cache/go-build \
    --mount=type=bind,target=. \
    CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
    go build -ldflags="-s -w" -o /out/server ./cmd/server

FROM gcr.io/distroless/static-debian12 AS runtime
COPY --link --from=build /out/server /server
USER 65532:65532
ENTRYPOINT ["/server"]

--mount=type=bind,target=. mounts the entire build context read-only for the compile step — no COPY layer at all, since the only output that matters is the binary in /out. The id=gobuild-${TARGETARCH} scoping keeps cross-platform builds from fighting over one build cache.

---

Anti-patterns

Anti-pattern	Why it breaks caching	Fix
COPY . . before installing dependencies	Every source edit re-installs everything	Copy manifests first
RUN apt-get update alone in a layer	Cached package index goes stale; installs fail or ship old packages	Fuse update + install in one RUN
ARG BUILD_DATE used near the top	Invalidates the entire build every run	Move metadata-consuming instructions last
.git not in .dockerignore	Context hash changes every commit	Add it
mode=min (or inline) on a multi-stage build	Intermediate stages never cached in CI	mode=max with a registry/gha backend
Cache mounts as the only CI strategy	Mounts don't survive ephemeral runners	Registry layer cache, or a persistent builder
One cache tag for all branches	Branches evict each other constantly	Scope per branch with a main fallback
Cache mount pointed at node_modules/target	Contents never land in the image	Cache the download dir; cp artifacts out in the same RUN
Secrets via ARG	Leaks into history and invalidates on rotation	--mount=type=secret
Merging every RUN into one giant chain "for fewer layers"	Any change re-runs all of it; layer count is not the metric	Split along volatility boundaries
ADD <url> without --checksum	Nondeterministic key controlled by a third party	Pin the checksum

That last one deserves emphasis: the old "minimize layer count" advice is largely obsolete. Layers are cheap, deduplicated, and pulled in parallel. Merging steps to save layers actively destroys cache granularity. Merge steps when they're logically atomic (update+install), not to hit a layer budget.

---

Checklist

Dockerfile

• # syntax=docker/dockerfile:1 on line one
• Base image pinned by digest, with automated bumps
• Instructions ordered least-volatile → most-volatile
• Dependency manifests copied and installed before source
• apt-get update fused with apt-get install
• Cache mounts on package-manager and compiler cache directories
• --link on final-stage COPY instructions (numeric --chown)
• Secrets via --mount=type=secret, never ARG
• Volatile labels and build args at the very bottom

Context

• .dockerignore covers .git, dependency dirs, build outputs, .env*
• transferring context is small — verify, don't assume

CI

• Builder driver is docker-container (or remote/cloud), not docker
• cache-to set with mode=max
• cache-from includes branch scope plus a default-branch fallback
• Multi-platform builds use per-platform cache scopes
• Cache backend storage is within quota and monitored

Verification

• Build twice with no changes → everything CACHED
• Edit one source file → dependency install stays CACHED
• Edit the lockfile → dependency install correctly re-runs
• Fresh CI run on a new branch → hits the main cache

Run those four verification builds after any Dockerfile change. A caching setup you haven't tested is a caching setup that isn't working — and the failure mode is silent, which is precisely why so many pipelines have been quietly rebuilding from scratch for months.