Repository navigation
🤖 tests: a digest-pinned bug-bash sandbox image, published manually from main - #5818
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
ThomasK33
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 94ef93353a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 14dc77b0ed
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Parked. Head The repeated mechanism: in
All open findings on this head (left open on purpose):
Two of them are not in the reuse path. One asks for a Make target for Evidence on this head:
Proposal (not started): remove the reuse path. Each authorized publish run builds, pushes by digest with no existence check, and attests only the output of its own build. Keep the The branch and the findings stay as they are. I will not push, add this PR to the merge queue or publish until there is a fresh explicit GO. Generated with |
ThomasK33
changed the title
…e digest (coder#5875) ## Summary This is PR 2 of 4 for the approved B1 plan (coder#5714). It adds the runner library of the bug-bash sandbox, plus `image.json` with the first published digest. The library has no entry point, so nothing can run a job yet. PR 3 adds staging, the mount checks, the app log and the launch command. ## Background coder#5818 added the image workflow, and a manual publish ran on `main`: [run 37759827317](https://github.com/coder/xum/actions/runs/37759827317) (`workflow_dispatch`, `main`, `752a624e15`). This PR records that digest in a reviewed file. That is the "a person opens the digest PR" step of the plan. It also adds the code that uses the digest. ## Implementation 1. `tests/bugbash/sandbox/image.json` pins the published image. I copied every value from tool output, not by hand: - The digest comes from the publish run's Attest step. It equals the sha256 of the registry manifest. - The key and the playwright-core version come from the image's labels. - `publishedFrom` is the run's head SHA. - The key equals `make bugbash-sandbox-key` on this branch. 2. `tests/bugbash/sandbox/runner.ts` has two parts. `readImageLock()` validates `image.json`. `Session` does the work: - **ensureImage()** first compares the checkout's inputs key (`build.sh --key`) with `image.json`. A mismatch refuses as a stale image, before any docker command. Next it finds the local daemon. If the image is missing, it pulls only `name@digest`. Then it checks that the image label `org.xum.bugbash.inputs` equals the key. An empty `docker images` list with exit 0 means "absent". Any query failure refuses, so a failure is never read as "absent". The runner never builds. - **Docker preflight** comes from coder#5802. It refuses non-Linux hosts, root, remote or ssh endpoints, Docker Desktop and rootless Docker. After `docker context inspect`, every docker command gets only `PATH`, `DOCKER_HOST` and an empty private `DOCKER_CONFIG`. - **Children:** every command is an async child that the session tracks. The entry point (PR 3) will abort the session's `AbortSignal` from its SIGINT and SIGTERM handlers. Then running children get SIGTERM, and SIGKILL after 5 s. Job commands started after that refuse with `Stopped`. - **cleanup(job)** is one idempotent promise for success, error and stop. It waits for every child. It removes only the container that matches the name and both labels (owner and checkout), and it confirms the removal. It reports `unknown: …`, never success, when it cannot read the container state. It removes the private client folder. It never removes an image. ## Validation 1. Gate, the anonymous pull: in a fresh `docker:27-dind` with no client config and no images, `docker pull ghcr.io/coder/xum-bugbash-sandbox@sha256:61adf1a543f2b1cc1373506556eefad52176aabd5bb1e491927eef82bde8fa73` exited 0 in 30,228 ms and gave 1,425,818,783 bytes. An anonymous `imagetools inspect` (empty `DOCKER_CONFIG`) gave the same digest, so the package is public. 2. Gate, the stale-key refusal: `runner.test.ts` "a stale inputs key refuses before any docker command". 3. `runner.test.ts` runs the real `build.sh` in throwaway git repos with a fake `docker`. It covers: - a pull by digest when the image is missing, and no pull when it is present; - a refusal on a label mismatch, on an image query failure, on a remote endpoint, with no daemon, on Docker Desktop and on rootless Docker; - the private client env; - a stop during a pull that ignores SIGTERM: it ends with `Stopped` after SIGKILL, new commands refuse, and cleanup still removes the job container with the full filter; - cleanup that reports `unknown` when `ps` fails, and that returns the same promise on a second call. 4. Mutation check: 10 mutants, all fail the tests. They are: no stale-key check, query failure read as absent, no label check, no SIGKILL escalation, cleanup gated by the stop, unknown reported as removed, user env passed to docker, cleanup not idempotent, remote endpoint allowed, and the checkout label filter dropped. 5. `make static-check` passes. `bun test ./tests/bugbash` passes (55 tests). Size: 3 files, +454 lines with tests. ## Risks Low. This PR adds test tooling only, and nothing calls the library yet. No file under `src/` changes. The coder#5815 host pause stays. Refs coder#5714 --- _Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$111.52`_ <!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high costs=111.52 -->
1 participant
Summary
This is PR 1 of 4 for the approved B1 plan (#5714). It adds a bug-bash sandbox image and the one script that builds it. Only a manual run of a CI workflow on
mainpublishes the image. No runner uses the image yet. PR 2 adds the runner library andimage.jsonwith the first digest.Background
The parked runner (#5802) built its image on each host, synchronously, before its signal handlers existed. The B1 plan replaces that build with an image that CI publishes. Runners pull it by the digest in a reviewed file. This PR covers only the image side.
Implementation
tests/bugbash/sandbox/Dockerfileinstalls tools only: node 22, bun 1.3.12, git, and Chromium for Playwright. Both base images are pinned by digest. The labelsorg.xum.bugbash.inputsandorg.xum.bugbash.playwright-corename its inputs.tests/bugbash/sandbox/build.shhas three modes, and the Make targets call it:make bugbash-sandbox-key(--key) prints the inputs key. The key is the sha256 of three values: the Dockerfile hash, the build.sh hash, and the playwright-core version of@e2e-dev/webinbun.lock(1.63.0, Chromium 1243). The root 1.57.0 copy does not count. The script refuses when an input is not committed or differs fromHEAD.make bugbash-sandbox-imagemakes a local linux/amd64 build and prints the image ID.make bugbash-sandbox-publish(--push) is for the workflow only. It always builds fresh: it never checks the registry and never reuses an image. It pushes a new tag,inputs-<key16>-<commit12>-<UTC time>, so a publish overwrites no tag. It prints theimage.jsonrecord, with the image name and digest taken from the build's own metadata.--no-cache --pull, no provenance or SBOM manifests, and no build args from the environment. The scripts work with the bash 3.2 that macOS ships, and they fall back toshasumwhensha256sumis missing..github/workflows/bugbash-sandbox-image.yml:permissions: {}.Makefileor the workflow runBuild (no push)withcontents: readonly, and report the size.workflow_dispatchrun onmain(github.ref == 'refs/heads/main'). It checks out that run's exactgithub.shawithout persisted credentials. It is the only job withpackages: write,id-token: writeandattestations: write. It pushes, writesimage.jsonto the job summary, and attests exactly the name and digest from thatimage.json. No cache and no artifact from another run is an input. Nopushtrigger exists.Nothing deletes an image. No file under
src/changes. The #5815 host pause stays.Publishing is manual (changes to the B1 plan)
main, then opens a pull request that copies the digest from the job summary intoimage.json. No workflow writes to the repo.bun.lock(the@e2e-dev/webplaywright-core version), the Dockerfile orbuild.shmerges, local runs refuse with the stale inputs-key error. They refuse until someone publishes a new image and its digest pull request merges.image.jsonstays the trust anchor.image.jsondigest and refuses when the image label differs from the checkout's key. PR 4's CI picks either theimage.jsondigest or a local candidate image. None of them looks up an image by its key or its tag.Package visibility (held for approval)
Validation
build.test.tsruns the realbuild.shin throwaway git repos:@e2e-dev/webcopy refuses.docker, two--pushruns of one key make two push builds and no registry call. They report the name and digest from the build metadata. A mutant that adds a registry lookup fails this test.make bugbash-sandbox-imageon this host: 45 s wall time, 1,425,909,942 bytes, linux/amd64. The labelorg.xum.bugbash.inputsequals the output ofmake bugbash-sandbox-key. A--network nonerun as uid 1000 found bun 1.3.12, node v22.23.3, git 2.39.5,chromium-1243andchromium_headless_shell-1243.Build (no push)on the previous head built 1,425,793,449 bytes.make static-check,make lint-actions(actionlint and zizmor) andbun test ./tests/bugbashpass (43 tests).Not measured yet: the GHCR pull time. It needs the first publish, which waits for approval.
Risks
The risk to the product is low. This PR changes only test tooling, three Make targets and a new workflow. The publish job is the only place with registry write access, and it runs only on a manual run on
main.Refs #5714
Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:high• Cost:$104.23