From a0352d9d16cce194a3c7a4718d20fe69648b0de2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Tue, 23 Jun 2026 17:05:01 +0300 Subject: [PATCH 1/2] docs: document enterprise testing coverage gates --- README.md | 2 + docs/enterprise-release-gates.md | 6 +++ docs/operator-testing.md | 84 ++++++++++++++++++++++++++++++++ universal-refiner/README.md | 2 +- 4 files changed, 93 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 513830d..0e3b334 100644 --- a/README.md +++ b/README.md @@ -58,6 +58,8 @@ flowchart LR - [Portfolio proof notes](./docs/portfolio-proof.md) - [Architecture spec](./docs/promptimprover-autogen-architecture-spec.md) +- [Operator testing guide](./docs/operator-testing.md) +- [Enterprise release gates](./docs/enterprise-release-gates.md) - [`universal-refiner/package.json`](./universal-refiner/package.json) - [`universal-refiner/tests`](./universal-refiner/tests) diff --git a/docs/enterprise-release-gates.md b/docs/enterprise-release-gates.md index f53c0c5..198a7df 100644 --- a/docs/enterprise-release-gates.md +++ b/docs/enterprise-release-gates.md @@ -2,6 +2,8 @@ Passing unit tests alone does not prove that PromptImprover is operationally ready. A release is eligible only when every required gate below passes. +For exact operator commands, current evidence, and the gate-to-CI mapping, see [Operator Testing Guide](./operator-testing.md). + ## Quality Target - All owned deterministic production logic reaches 100% statements, branches, functions, and lines. @@ -39,6 +41,10 @@ Initial high-risk gaps include MCP dispatcher behavior, background autonomy, tem The enforced ratchet is 100% statements, branches, functions, and lines. It cannot be lowered without an approved exception. +## Current Verified Release Baseline + +The current release gate baseline is documented in [Operator Testing Guide](./operator-testing.md#current-verified-baseline). At the time this page was updated, `master` commit `abbff59cc6d62b734912e2a98c61ae3dc1d4c6b8` had a successful GitHub CI run (`28030976193`) and a local `npm.cmd run release:verify` pass with 51 test files, 382 tests, and 100% statements, branches, functions, and lines. + ## Operator Recovery Build before invoking the recovery commands. Both operations run SQLite integrity checks and fail closed: diff --git a/docs/operator-testing.md b/docs/operator-testing.md index 708a732..7e07199 100644 --- a/docs/operator-testing.md +++ b/docs/operator-testing.md @@ -2,6 +2,69 @@ Use this guide to verify PromptImprover from a clean Windows operator session. +## Current Verified Baseline + +This is the latest known-good baseline at the time this guide was updated: + +| Evidence | Value | +|---|---| +| Date | 2026-06-23 | +| Branch | `master` | +| Commit | `abbff59cc6d62b734912e2a98c61ae3dc1d4c6b8` | +| GitHub CI run | `28030976193` | +| CI result | `success` | +| Local release gate | `npm.cmd run release:verify` passed | +| Local test count | 51 test files, 382 tests | +| Coverage | 100% statements, branches, functions, and lines | +| Local runtime health | `/api/health` returned `runtime.status: online` | +| Local semantic provider | `http://localhost:11434`, models `gemma3:12b` and `gemma3` | + +Treat this table as evidence, not a permanent guarantee. When any product behavior changes, rerun the gate and update this baseline. + +## Coverage Policy + +Coverage is enforced by `universal-refiner/vitest.config.ts`: + +| Metric | Required | +|---|---:| +| Statements | 100% | +| Branches | 100% | +| Functions | 100% | +| Lines | 100% | + +The coverage include set is owned deterministic production logic under: + +- `hooks/lib/**/*.ts` +- `src/**/*.ts` + +The only current exclusion is generated version metadata: + +- `src/core/generated-version.ts` + +Do not lower coverage thresholds to merge a feature. If a defect is reproduced, add a regression test at the owning boundary before fixing or merging. + +## Release Gate Matrix + +| Gate | Command | Scope | CI job | +|---|---|---|---| +| Clean install | `npm ci` | Dependency graph from lockfile | All jobs | +| Build | `npm.cmd run build` | TypeScript and dashboard copy | All jobs | +| Coverage | `npm.cmd run test:coverage` | Unit and integration tests with 100% thresholds | `build-and-test`, `windows` | +| MCP acceptance | `npm.cmd run test:acceptance` | Advertised MCP tool schemas and dispatcher paths | `acceptance`, `windows` | +| Semantic fallback | `npm.cmd run acceptance:semantic` | local provider ordering, fallback, malformed response, timeout, outage | `acceptance`, `windows` | +| Tracked turn | `npm.cmd run acceptance:tracked-turn` | prompt ID and SQLite outcome linkage | `acceptance`, `windows` | +| Stress tests | `npm.cmd run test:stress` | concurrent and long-running behavior | `stress`, `windows` | +| EventStore stress | `npm.cmd run stress:event-store` | SQLite contention and multi-process behavior | `stress`, `windows` | +| Abrupt recovery | `npm.cmd run recovery:event-store:abrupt` | interrupted writer recovery | `stress`, `windows` | +| Soak | `npm.cmd run stress:event-store:soak` | long-duration EventStore behavior | `stress`, `windows` | +| Production audit | `npm.cmd run security:audit` | production dependency vulnerabilities, high or above | `supply-chain` | +| Full audit | `npm.cmd run security:audit:all` | production and development dependency vulnerabilities, high or above | `supply-chain` | +| Secret scan | `npm.cmd run security:secrets` | committed credential patterns | `supply-chain` | +| Package dry-run | `npm.cmd run package:check` | npm package contents | `supply-chain` | +| Package runtime | `npm.cmd run acceptance:package-runtime` | packed tarball install plus `/api/health` smoke | `supply-chain`, `windows` | +| Release gate | `npm.cmd run release:verify` | local aggregate of the gates above | local operator | +| CI release gate | GitHub Actions `release-gate` job | all enterprise jobs must pass before merge | `release-gate` | + ## 1. Enter The Active Package ```powershell @@ -27,6 +90,8 @@ Expected result: - Package dry-run passes. - `acceptance:package-runtime` installs the packed tarball into a temporary global prefix and serves `/api/health`. +If this command fails, do not bypass it. Fix the failing behavior or document an explicit, reviewed exception in this file and in `docs/enterprise-release-gates.md`. + ## 3. Check Global MCP Registration ```powershell @@ -151,3 +216,22 @@ Expected result: - `release-gate` passes. Remote CI is the authoritative proof for Linux and Windows clean-checkout behavior. + +## 9. Document New Tests + +Every production feature or bug fix should update this guide when it changes how the product is verified. + +Use this checklist: + +- Add or update tests at the smallest useful boundary. +- Add acceptance or stress coverage for cross-process, CLI, MCP, SQLite, or packaging behavior. +- Update the release gate matrix when a new script becomes part of the release contract. +- Update the current verified baseline after the branch is merged and CI is green. +- Keep limitations explicit. A green gate proves declared checks passed; it does not prove unknown future failures are impossible. + +## Known Limitations + +- Live Gemma verification depends on a local or external OpenAI-compatible model endpoint. The deterministic release gate covers provider fallback without requiring the operator's live model server. +- MCP tool availability depends on the hosting client exposing a healthy MCP transport. If a live `lint_prompt` or refinement call closes its transport, restart the MCP runtime and rerun the global registration doctor before treating the CLI as healthy. +- External CLI hook behavior depends on each client supporting hooks and trusting the current workspace. Codex currently uses MCP-first operation rather than transparent prompt lifecycle interception. +- "100% coverage" means every currently included deterministic production line, branch, statement, and function is covered. It does not mean every possible integration, environment, timing, or future regression is impossible. diff --git a/universal-refiner/README.md b/universal-refiner/README.md index a75fa49..d104872 100644 --- a/universal-refiner/README.md +++ b/universal-refiner/README.md @@ -101,7 +101,7 @@ See `.universal-refiner.example.json` for an annotated template. npm run release:verify ``` -Runs build, 100% test coverage, MCP acceptance, semantic fallback, stress/soak, and audit checks. +Runs build, 100% test coverage, MCP acceptance, semantic fallback, stress/soak, audit checks, package dry-run, and package-runtime smoke testing. See the repo-level [operator testing guide](../docs/operator-testing.md) and [enterprise release gates](../docs/enterprise-release-gates.md) for the full release contract. ## Security From 7eaad612b5b3bc2147790fdfd3fc790b59a4c69a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Tue, 23 Jun 2026 17:07:42 +0300 Subject: [PATCH 2/2] fix: patch legacy mcp-server qs advisory --- mcp-server/package-lock.json | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/mcp-server/package-lock.json b/mcp-server/package-lock.json index cb37244..3050fa7 100644 --- a/mcp-server/package-lock.json +++ b/mcp-server/package-lock.json @@ -195,6 +195,7 @@ "resolved": "https://registry.npmjs.org/@types/node/-/node-25.5.2.tgz", "integrity": "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg==", "license": "MIT", + "peer": true, "dependencies": { "undici-types": "~7.18.0" } @@ -585,6 +586,7 @@ "resolved": "https://registry.npmjs.org/express/-/express-5.2.1.tgz", "integrity": "sha512-hIS4idWWai69NezIdRt2xFVofaF4j+6INOpJlVOLDO8zXGpUVEVzIYk12UUi2JzjEzWL3IOAxcTubgz9Po0yXw==", "license": "MIT", + "peer": true, "dependencies": { "accepts": "^2.0.0", "body-parser": "^2.2.1", @@ -789,6 +791,7 @@ "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.26.tgz", "integrity": "sha512-uyZtpnYxM9CmQ7QsQknM4zN8EftNqhON1qYeIKM0Se67CCEe2c44xyGURwB0axX2fBDu1dqHrHAc1hmNT8ITkw==", "license": "MIT", + "peer": true, "engines": { "node": ">=16.9.0" } @@ -1056,9 +1059,9 @@ } }, "node_modules/qs": { - "version": "6.15.0", - "resolved": "https://registry.npmjs.org/qs/-/qs-6.15.0.tgz", - "integrity": "sha512-mAZTtNCeetKMH+pSjrb76NAM8V9a05I9aBZOHztWy/UqcJdQYNsf59vrRKWnojAT9Y+GbIvoTBC++CPHqpDBhQ==", + "version": "6.15.2", + "resolved": "https://registry.npmjs.org/qs/-/qs-6.15.2.tgz", + "integrity": "sha512-Rzq0KEyX/w/tEybncDgdkZrJgVUsUMk3xjh3t5bv3S1HTAtg+uOYt72+ZfwiQwKdysThkTBdL/rTi6HDmX9Ddw==", "license": "BSD-3-Clause", "dependencies": { "side-channel": "^1.1.0" @@ -1351,6 +1354,7 @@ "integrity": "sha512-bGdAIrZ0wiGDo5l8c++HWtbaNCWTS4UTv7RaTH/ThVIgjkveJt83m74bBHMJkuCbslY8ixgLBVZJIOiQlQTjfQ==", "dev": true, "license": "Apache-2.0", + "peer": true, "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" @@ -1426,6 +1430,7 @@ "resolved": "https://registry.npmjs.org/zod/-/zod-4.3.6.tgz", "integrity": "sha512-rftlrkhHZOcjDwkGlnUtZZkvaPHCsDATp4pGpuOOMDaTdDDXF91wuVDJoWoPsKX/3YPQ5fHuF3STjcYyKr+Qhg==", "license": "MIT", + "peer": true, "funding": { "url": "https://github.com/sponsors/colinhacks" }