Skip to content

build: update Docker image for eMudhra/emSign CA incident - #13374

Merged
julian-risch merged 1 commit into
mainfrom
fix/docker-emsign-ca-certs
Oct 21, 2026
Merged

julian-risch merged 1 commit into
mainfrom
fix/docker-emsign-ca-certs

Conversation

@bogdankostic

Copy link
Copy Markdown
Contributor

Related Issues

Proposed Changes:

All 8 eMudhra/emSign CAs had trivially weak private keys, so certificates they sign can be forged. Our Docker image trusted them in three places; this PR fixes all three:

  • Base image: bump the python:3.12-slim digest in Dockerfile.base and docker-bake.hcl. The new image ships Debian's ca-certificates 20250419+deb13u1 (DSA-6572-1), which drops emSign ECC Root CA - G3. The old pin was built on 2026-05-19, so this also brings five months of Debian patch updates, including Python 3.12.13 → 3.12.15 and OpenSSL 3.5.6 → 3.5.7.
  • pip in /opt/venv: python3 -m venv installs the pip bundled with CPython (25.0.1), and nothing upgraded it so far. We now install pip>=26.2.2. The floor makes the build fail instead of silently installing an affected pip, e.g. while our 24h exclude-newer cooldown still hides 26.2.2.
  • uv in the build stage: 0.11.17 → 0.13.4. uv embeds its own root certificates and downloads every package that goes into the image.

How did you test it?

Built the image locally for linux/arm64 and checked inside it:

  • pip in the venv: 26.2.2 (was 25.0.1)
  • certifi and pip's vendored certifi: both 2026.10.19; emSign ECC Root CA - G3 is not in the bundle
  • ca-certificates: 20250419+deb13u1 (was 20250419, which included emSign_ECC_Root_CA_-_G3.pem)
  • from haystack.version import __version__ works (3.5.0-rc0)

Until 2026-10-21 16:36 UTC the build fails on purpose, because the cooldown still hides pip 26.2.2 (confirmed locally). The checks above ran on an otherwise identical copy with --exclude-newer-package pip=false.

To reproduce:

cd docker
IMAGE_TAG_SUFFIX=test docker buildx bake -f docker-bake.hcl base --set base.platform=linux/arm64 --load
docker run --rm deepset/haystack:base-test sh -c 'pip --version; python -c "import certifi; print(certifi.__version__)"; dpkg-query -W ca-certificates'

Notes for the reviewer

  • Please merge after 2026-10-21 16:36 UTC. Before that, the Docker release job on main fails on the pip floor.
  • Merging only republishes base-main. stable and base-v3.4.0 keep the old image, and re-running the release workflow on the v3.4.0 tag would rebuild from that tag's old Dockerfile. Proposal: cherry-pick onto v3.4.x and release v3.4.1, or ship it with v3.5.0.
  • The base image's own pip in /usr/local stays at 25.0.1 until a CPython 3.12 release bundles a fixed pip. /opt/venv comes first on PATH, so pip and python -m pip use the fixed one. Dependabot doesn't track the ARG digests, so that follow-up bump has to be done by hand.
  • No release note: Docker-only change, same as ci: pin container images by digest #11460.

Checklist

Bump the python:3.12-slim digest to pick up Debian's ca-certificates
20250419+deb13u1 (DSA-6572-1), bump the build-stage uv to 0.13.4, and
upgrade pip in /opt/venv to >=26.2.2, since the pip bundled with CPython
still trusts the compromised CAs.

See https://discuss.python.org/t/109576

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bogdankostic
bogdankostic requested a review from a team as a code owner October 21, 2026 15:43
@bogdankostic
bogdankostic requested review from anakin87 and removed request for a team October 21, 2026 15:43
@vercel

vercel Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
haystack-docs Ignored Ignored Preview Oct 21, 2026 4:04pm UTC

Request Review

@julian-risch
julian-risch merged commit a7e79f3 into main Oct 21, 2026
24 checks passed
@julian-risch
julian-risch deleted the fix/docker-emsign-ca-certs branch October 21, 2026 17:54
julian-risch pushed a commit that referenced this pull request Oct 21, 2026
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants