The Digest Is the Container. The Tag Is Just a Rumor.

Chasing a model that behaves differently across environments?

I agree to the Privacy Policy Talk ML Infra

I’ve watched a product-ranking model reorder the same search results differently in staging than it did in the notebook — same commit, same requirements.txt, same “it works on my machine” — and spent an afternoon convinced I was losing my mind before I found the actual diff: a different BLAS backend, three layers down in the dependency tree, that pip freeze had no way of ever telling me about.

That’s the failure mode this piece is about, and you don’t have to take my word for it. Run this:

python3 -c "import numpy as np; np.show_config()"

On one machine you’ll see openblas. On another, mkl. Both satisfy numpy==1.26.4 in your requirements file. Neither shows up in pip freeze, because a wheel’s linked math library isn’t a Python-level dependency — it’s baked in at build time, and which wheel you got depends on your platform, your pip version, and the day you ran pip install. Floating-point addition isn’t associative, BLAS backends don’t sum in the same order, and after a few million gradient updates that “irrelevant” implementation detail is a different set of model weights.

The spine underneath everything below: a container’s actual identity is its digest — a hash of every layer, all the way down to libc — not the tag on the label. Tags drift. Digests don’t. Every argument in this piece cashes out against that one fact.

Here’s the plan, in order: what a tag actually is versus what a digest is; why that distinction is a derivable consequence of how images are built, not a rule someone made up; why a venv can’t give you the same guarantee no matter how hard you pin it; one training image built end to end, digest-pinned and multi-staged; why the GPU case makes all of this worse, not better; where the guarantee still doesn’t reach even inside a container; and, finally, where Kubernetes actually picks up — because it picks up later than most people think.

The tag is a pointer, the digest is a promise

Pull a base image and ask Docker what it actually gave you:

docker pull python:3.11-slim
docker inspect python:3.11-slim --format='{{index .RepoDigests 0}}'
# python:3.11-slim@sha256:2fadb8bc1a4b3f...

python:3.11-slim is a label the Python image maintainers move whenever they rebuild — a Debian security patch, a glibc bump, a new pip. Same tag, different bytes, and there’s no changelog notification when it happens under you. The line after it, the @sha256:..., is the thing that can’t move. Pin that instead:

docker pull python:3.11-slim@sha256:2fadb8bc1a4b3f...

Now the base of your image is the same base every time, on every machine, forever — or until you deliberately change the pin.

Why the digest is actually load-bearing, not just paranoid

This isn’t an arbitrary rule; it falls out of how an image is built. Every Dockerfile instruction that changes the filesystem produces a layer, and each layer is content-addressed — its identifier is a hash of its own contents. The image manifest lists those layer hashes in order, and the image digest is a hash of that manifest. Change one byte in one layer, anywhere in the stack, and every hash above it changes too — a Merkle tree, the same structure Git commits and blockchains use for the same reason.

That’s why pinning the digest pins the entire transitive closure of what’s inside the image, not just the packages you thought to list. A libc patch three layers below your pip install line changes the digest exactly as loudly as changing your own code would. requirements.txt can only describe the layer it lives in. The digest describes all of them, whether you remembered to think about them or not.

Why not just a venv, then

A venv is lighter, faster to create, and doesn’t need a daemon running in the background — all real, and none of it a reason to skip this section. If your only dependency surface is pure-Python packages, a venv with a hash-locked requirements.txt (pip-compile --generate-hashes, or uv pip compile) genuinely gets you most of the way, for less cost. Say that part out loud, because it’s true.

Here’s where it stops being true: a venv pins packages inside an already-running Python interpreter, on top of a system it has no opinion about. It cannot pin the Python interpreter’s own build, the system BLAS library, the CUDA driver and runtime, or libc — because none of those are Python packages. They’re the floor the venv is standing on, and pip freeze has never once looked at the floor.

# requirements.txt pins the package. It says nothing about what it links against.
ldd $(python3 -c "import numpy.core._multiarray_umath as m; print(m.__file__)") | grep -i blas
# libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x00007f...)

A container image is a full filesystem snapshot, floor included. That’s the entire difference: a venv pins the packages, an image pins the world the packages run in.

Building one reproducible training image, end to end

One example, all the way through — a small scikit-learn job that trains an e-commerce product-ranking model, containerized properly.

# syntax=docker/dockerfile:1
FROM python:3.11-slim@sha256:2fadb8bc1a4b3f... AS builder

WORKDIR /build
COPY requirements.txt .
RUN pip install --no-cache-dir --user -r requirements.txt

FROM python:3.11-slim@sha256:2fadb8bc1a4b3f...

RUN useradd --create-home --uid 1000 trainer
COPY --from=builder /root/.local /home/trainer/.local
COPY train.py .

USER trainer
ENV PATH="/home/trainer/.local/bin:${PATH}"
ENTRYPOINT ["python", "train.py"]

Two stages, two different jobs. The builder stage has pip, wheel build tooling, and everything else needed to install dependencies — none of which the running container needs at 2 a.m. in production. The final stage copies only the installed packages across and drops the build tooling on the floor, which is also why it doesn’t run as root: nothing in the runtime stage needs the privilege, so nothing gets it.

docker build -t train-product-ranker:$(git rev-parse --short HEAD) .
docker images train-product-ranker

The size difference is the same reason python:3.11-slim (roughly 130 MB) exists next to plain python:3.11 (north of 1 GB): the full image ships a C compiler and header files your running container will never call. A single-stage build that apt-get install build-essentials and never cleans up pays that tax forever. Multi-stage pays it once, at build time, and ships none of it.

Tag by commit, not by latest — latest is a tag like any other, which means it’s exactly as mutable as python:3.11-slim was two sections ago, and for the identical reason.

The GPU case makes this worse, not better

CPU images have one moving floor: libc, BLAS, the interpreter. GPU images have a second one stacked on top — the CUDA driver on the host has to match the CUDA runtime baked into the image, which has to match the PyTorch or TensorFlow build you installed, and none of those three are the same artifact.

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Run that before anything model-specific loads. If it fails, the problem is the GPU passthrough itself, not your code — and you want to know that in ten seconds, not after twenty minutes of chasing a stack trace that blames torch. If it succeeds, you’ve confirmed the one thing a CPU image never had to prove: that the container can actually see the device it thinks it has.

FROM nvidia/cuda:12.4.1-runtime-ubuntu22.04@sha256:9a1b2c3d...

RUN apt-get update && apt-get install -y --no-install-recommends python3-pip \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# torch==2.4.0+cu124 in requirements.txt — the +cu124 has to agree with the base image tag,
# or you get a working import and a runtime error the first time a tensor touches the GPU.

The +cu124 suffix on the torch wheel and the 12.4.1 in the base image tag are the same fact, stated twice, in two different files, by two different teams who have never met. Nothing checks that they agree except you, at build time, on purpose.

What a container still doesn’t give you

Two honest limits, both worth stating plainly rather than discovering in production.

First: Docker content-addresses the output of a build, not its inputs. RUN apt-get install -y curl with no version pin can resolve to a different curl build tomorrow even with an unchanged Dockerfile, because apt-get resolves against whatever’s in the package mirror right now. The digest you get at the end is still real and still pinnable — but the build that produced it wasn’t guaranteed reproducible, only its result was. Nix takes the harder, more expensive road here: it hashes the entire build recipe — every input, every flag, every dependency version — into a derivation hash before the build runs, so two machines evaluating the same expression are byte-identical by construction, not by luck. Most ML teams don’t reach for Nix anyway, and the reason is boring rather than technical: CUDA base images, prebuilt PyTorch wheels, and Kubernetes’s own scheduler all assume OCI images as the unit of deployment. Nix buys a stronger guarantee than most training pipelines currently need, at an adoption cost most teams aren’t signed up for.

Second: pinning the image doesn’t make GPU training deterministic. Some cuDNN convolution algorithms pick the fastest available kernel per run, and “fastest” isn’t always the same kernel twice, container or not.

import torch

torch.use_deterministic_algorithms(True)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

That costs you throughput — the whole reason cudnn.benchmark exists is to let it pick a faster, less predictable kernel. The container fixes the software world underneath the run. It was never going to fix the algorithm’s own coin flip.

Where Kubernetes actually starts

A container answers one question: does this exact process run identically, wherever I run it. That’s a single-machine, single-process guarantee, and everything above this line was in service of making that guarantee actually true instead of just assumed.

Kubernetes answers a different question, one that doesn’t even make sense until the first one’s already settled: given N copies of this image, which of M machines runs how many, what happens when one dies, and how do the others find it. Scheduling, restart policy, service discovery — none of that is downstream of code quality. It’s downstream of having correctly answered “will this image behave the same way twice,” because Kubernetes doesn’t check your digest for you. It schedules whatever digest you gave it, correct or not, and it will restart a silently-wrong container exactly as reliably as it restarts a correct one. Orchestration doesn’t fix a reproducibility bug. It reproduces it, on schedule, across every replica.

That’s the actual order of operations, and it’s easy to get backwards: reach for a Kubernetes manifest before the image underneath it is trustworthy, and you’ve automated the deployment of a coin flip.

What to do Monday morning

IMAGE := train-product-ranker
TAG   := $(shell git rev-parse --short HEAD)

build:
	docker build -t $(IMAGE):$(TAG) .

digest:
	docker inspect $(IMAGE):$(TAG) --format='{{index .RepoDigests 0}}'

verify-gpu:
	docker run --rm --gpus all $(IMAGE):$(TAG) \
		python -c "import torch; print(torch.cuda.is_available())"

Four habits, in priority order: pin base images by digest, not tag — grep your Dockerfiles for a bare FROM today and you’ll probably find one. Split builder from runtime and drop the compiler you don’t ship. Tag by commit, never latest. Run the GPU check as its own step, before the training script gets a chance to make the failure look like yours.

Closing

Kubernetes gets the credit, because it’s the thing you write YAML for and stare at dashboards about. The container is the thing you stopped noticing, the same way nobody thanks TCP for arriving in order. ML engineering didn’t invent the reproducibility problem — dependency drift is as old as make. It just got expensive enough, in GPU-hours and in a ranking model that quietly reordered results differently in staging than it did in the notebook, that going back and reading the Dockerfile stopped being optional.

Chasing a model that behaves differently across environments?

I agree to the Privacy Policy Talk ML Infra