Skip to content

🤖 fix: make no provider model calls under XUM_MOCK_AI - #5630

Merged
ThomasK33 merged 2 commits into
mainfrom
fix/5604-mock-ai-background-calls
Oct 4, 2026
Merged

ThomasK33 merged 2 commits into
mainfrom
fix/5604-mock-ai-background-calls

Conversation

@ThomasK33

Copy link
Copy Markdown
Member

Summary

With XUM_MOCK_AI=1, model creation now refuses before it builds a provider model, so background features (workspace title, sidebar status, branch summary, memory consolidation and harvest, continuous compaction summary, refine, workflow evaluate, auto model routing) make no provider request. Chat turns still play through the mock stream player, which creates no models.

Fixes #5604

Background

Mock mode replaced only the chat turn player. Every other model caller still created a real model from providers.jsonc, so a mock-mode server sent claude-haiku requests from the status timer and title generation. The remote UAT for #5602 saw these as repeated AI_RetryError logs.

Implementation

Every inference caller in src/node gets its model from one of three functions. A read-only inventory of all generateText/streamText/evaluate callers found no other path. The perf replay mode (XUM_REPLAY_TAPES) already refuses at the same three points, and this change adds the mock check next to it:

  1. ProviderModelFactory.createModelEffect: chat, title, status, summary, memory, compaction summary, refine, advisor and intuition models. It returns { type: "unknown", raw: "Model calls are disabled in mock AI mode (XUM_MOCK_AI=1)" }.
  2. ProviderModelFactory.createEvaluationModelEffect: workflow evaluate() and intuition evaluation.
  3. evaluationModelFactory.createEvaluationModel: auto model routing, which does not use ProviderModelFactory.

isMockAiMode() (new src/node/services/mock/mockAiMode.ts) reads XUM_MOCK_AI (or legacy MUX_MOCK_AI). AIService now uses it too, so the flag has one definition. mockAiStreamPlayer.ts is unchanged.

Decision (conservative option): refuse instead of routing background generators through a mock model. A mock model would make memory consolidation write mock output into memory files. Callers already handle a refused creation, as they do for a missing API key: titles fall back, and status generation retries on its normal schedule with a debug log.

Scope note: the server makes some network calls that are not inference, and this change leaves them alone. Model discovery runs on user request. Coder and Copilot catalog refreshes run after OAuth login. Voice transcription runs when the user records.

Validation

src/node/services/mockAiModelGate.test.ts points the anthropic and openai providers at a loopback HTTP server that records every request.

  1. The control case (no mock env) shows the title generator reaching the loopback server, and every model entry point succeeding.
  2. The mock case sets XUM_MOCK_AI=1. It runs the title and status generators end to end and calls every model entry point (including the auto-routing factory). It then asserts the loopback server received no request.

Pre-fix result of this test on origin/main (the status generator reached the provider):

129 |     // Background generators, end to end.
130 |     const title = await generateWorkspaceIdentity("Fix the login bug", [{ model: MODEL }], service);
131 |     expect(title.success).toBe(false);
132 |     const status = await generateWorkspaceStatus("user: fix the login bug", [MODEL], service);
133 |     expect(status.success).toBe(false);
134 |     if (!status.success) expect(status.error.reachedProvider).toBe(false);
                                                                    ^
error: expect(received).toBe(expected)
Expected: false
Received: true
(fail) XUM_MOCK_AI model gate > with XUM_MOCK_AI=1 no background model caller reaches the provider [18.00ms]
 1 pass
 1 fail

Risks

Low, and limited to mock mode. E2E runs set XUM_MOCK_AI=1 but configure no real provider, so their title and status generation already failed at model creation. In-process tests that call enableMockMode() without the env var are unaffected.


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-04T19:01:30.097538Z afae467 PR opened
🔒 Security Review ✅ Completed 2026-10-04T19:03:15.474285Z afae467 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@ThomasK33

Copy link
Copy Markdown
Member Author

Readiness record:

  • Final commit: afae4677426a2f5fdd48f42ec583d576580fb8ad
  • CI: Required passed on this commit. Smoke / Docker was skipped by its path filter.
  • Reviews: 3 of 6 assessments used (automatic normal review, automatic security review, final independent check). Both automatic reviews had no findings.
  • Final independent check: READY.
  • Deferred: voice transcription gate and the enableMockMode() / env-gate mismatch, tracked in 🤖 fix: close the remaining XUM_MOCK_AI provider-call gaps (voice, enableMockMode) #5632.
  • Decision: ready. Merging with --match-head-commit.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

@ThomasK33
ThomasK33 added this pull request to the merge queue Oct 4, 2026
Merged via the queue into main with commit 65c6b2c Oct 4, 2026
34 checks passed
@ThomasK33
ThomasK33 deleted the fix/5604-mock-ai-background-calls branch October 4, 2026 19:25
yermakoffivan pushed a commit to yermakoffivan/mux that referenced this pull request Oct 5, 2026
…t on image PRs (coder#5627)

## Summary

The Docker server image now ships `dist/runtime/analyticsWorker.js` and
the DuckDB native bindings it needs, so the analytics worker starts in
Docker. `Smoke / Docker` now also runs on PRs that touch server image
inputs, and it starts the shipped analytics worker instead of checking
only `/health`.

Fixes coder#5603
Fixes coder#5605

## Background

`AnalyticsService` starts `analyticsWorker.js` from the directory of the
running server bundle. `make build-docker-runtime` never built that
file, so every analytics request in the Docker image failed with `Cannot
find module '/app/dist/runtime/analyticsWorker.js'`. `/health` does not
start the worker (it starts lazily), and `Smoke / Docker` ran only in
the merge queue, so nothing caught this.

The two issues share one PR on purpose: the new smoke step is the
regression test for coder#5603, and this PR touches the `Dockerfile`, so its
own CI run shows the new trigger.

## Implementation

1. `Makefile`: a new `dist/runtime/analyticsWorker.js` target bundles
the worker with esbuild. `@duckdb/*` stays external because esbuild
cannot bundle the native `.node` binding.
`verify-docker-runtime-artifacts` now checks the file.
2. `Dockerfile`: the runtime stage copies `node_modules/@duckdb`. The
builder stage first deletes the musl bindings, because the runtime image
is glibc (`node:22-slim`). `detect-libc`, which the bindings use, is
already in the image for sharp.
3. `scripts/check-analytics-worker.cjs`: starts a built worker, runs its
`init` task (opens a DuckDB database through the native binding), then
shuts it down. It exits nonzero on any worker error.
4. `.github/workflows/pr.yml`:
- A new `docker` path filter in `changes` lists everything the
Dockerfile copies into the build stage (`src/**`, `docs/**`, package,
lock and patch files, the Makefile, the copied scripts, the Vite entry
files, `public/**`, `static/**`), plus `Dockerfile`, `.dockerignore` and
`pr.yml` itself. A Dockerfile comment asks to keep the two lists in
sync.
- `Smoke / Docker` runs on `pull_request` when that filter matches.
Merge queue and push behavior is unchanged.
- The container test runs the check script inside the image: `docker
exec -i mux-test node - /app/dist/runtime/analyticsWorker.js <
scripts/check-analytics-worker.cjs`.

Decision (conservative option): ship the worker instead of disabling
analytics in Docker. The image grows by about 75 MB (436 MB to 511 MB
locally) for `libduckdb.so`.

## Validation

I built the image from `origin/main` (before) and from this branch
(after) with `docker build`, ran each container, started the worker with
the check script, and called `POST /api/analytics/getSummary` with the
container's auth token.

<details>
<summary>Before (origin/main)</summary>

```
$ curl /health
{"status":"ok"}
$ docker exec ws72-before ls dist/runtime
mcpIconDecode.js
server-bundle.js
tokenizer-encoding-cl100k_base.js
tokenizer-encoding-claude.js
tokenizer-encoding-o200k_base.js
tokenizer-encoding-p50k_base.js
tokenizer.worker.js
$ docker exec -i ws72-before node - /app/dist/runtime/analyticsWorker.js < scripts/check-analytics-worker.cjs
analytics worker error: Cannot find module '/app/dist/runtime/analyticsWorker.js'
exit=1
```

Server log after `POST /api/analytics/getSummary` (response:
`INTERNAL_SERVER_ERROR`):

```
6:44.789PM dist/runtime/server-bundle.js:27 [AnalyticsService] Worker error { error: "Cannot find module '/app/dist/runtime/analyticsWorker.js'" }
6:44.791PM dist/runtime/server-bundle.js:27 ORPC /api/analytics/getSummary: Error: Cannot find module '/app/dist/runtime/analyticsWorker.js'
6:44.793PM dist/runtime/server-bundle.js:27 [AnalyticsService] Worker exited unexpectedly { code: 1 }
```

</details>

<details>
<summary>After (this branch)</summary>

```
$ curl /health
{"status":"ok"}
$ docker exec ws72-after ls dist/runtime node_modules/@duckdb
/app/dist/runtime:
analyticsWorker.js
mcpIconDecode.js
server-bundle.js
tokenizer-encoding-cl100k_base.js
tokenizer-encoding-claude.js
tokenizer-encoding-o200k_base.js
tokenizer-encoding-p50k_base.js
tokenizer.worker.js

/app/node_modules/@duckdb:
node-api
node-bindings
node-bindings-linux-x64
$ docker exec -i ws72-after node - /app/dist/runtime/analyticsWorker.js < scripts/check-analytics-worker.cjs
[analytics-worker] Shutting down, closing DuckDB
analytics worker OK: /app/dist/runtime/analyticsWorker.js
exit=0
```

```
$ POST /api/analytics/getSummary
{"totalSpendUsd":0,"todaySpendUsd":0,"avgDailySpendUsd":0,"cacheHitRatio":0,"totalTokens":0,"totalResponses":0}
$ docker logs ws72-after | grep -i analytics
[analytics-worker] syncCheck: plan=noop, workspacesOnDisk=0, watermarksInDB=0 (21ms)
```

</details>

Smoke / Docker trigger:
- This PR touches `Dockerfile`, so the job ran on its first head (pass,
and the log shows `analytics worker OK`):
https://github.com/coder/xum/actions/runs/37226228085/job/111506356866
- coder#5630 touches no image input at its first head (it changed only
`src/node/**`, before this PR widened the filter to `src/**`), so the
job was skipped there (skipping):
https://github.com/coder/xum/actions/runs/37226538034/job/111507277313

## Risks

Low. The change adds files to the image and a CI job to most code PRs.
`Smoke / Docker` took about 4 minutes on this PR, in parallel with the
other jobs. A new Dockerfile COPY input that is missing from the filter
reaches the merge queue unchecked, as before.

---

_Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking:
`high`_

<!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high
costs= -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🤖 fix: XUM_MOCK_AI still sends background claude-haiku requests

1 participant