APTL should own the observability stack as backend evidence-capture infrastructure

[Brad-Edwards]

Finding

APTL has no observability infrastructure of its own. The OpenTelemetry stack that serves evidence capture — collector, trace store, and the operator view over it — is currently declared by the TechVault scenario pack, as three nodes named aptl-otel-collector, aptl-tempo and aptl-grafana-otel.

That is the wrong owner. A scenario declares what the exercise happens in and what it needs captured; how capture is performed, where it is stored, and whether any of it is exposed to an operator are the backend's. The pack is being asked to remove them in OpenRAE/env-packs#337, and APTL needs to own them before that lands.

Why this is not just "move three containers"

The scenario's side of the contract is evidence_requirements, which states what must be captured without naming what captures it — scope, channel, media types, sensitivity, redaction, integrity, retention, loss disclosure. TechVault already declares one:

evidence_requirements:
  cortex-job-index-schema-readback:
    scope: fresh native schema readback projected to portable field semantics
    channel: api_response
    sensitivity: plain
    redaction: redact_secrets
    retention: run_lifetime
    loss_disclosure: required

So the deliverable is capture infrastructure that satisfies declared requirements of that shape, not a fixed Grafana deployment that happens to exist. Whatever APTL stands up should be answerable to those fields — particularly redaction, retention and loss_disclosure, which are promises about evidence rather than deployment details.

ADR-012 already frames the OTel stack as "always-on lab infrastructure (not optional)", and CORE_PROFILES = ("otel",) already marks the profile backend-owned and always selected. The ownership is stated; what is missing is realization that does not depend on a pack declaring the nodes.

The gap that makes sequencing matter

With an env-pack, the realized service set is derived from the pack. As raes_profiles.py records it: "An env-pack never reaches here: it ships no docker-compose.yml, so load_compose_profile_index returns an empty index (ADR-048) and node service identity is derived from the realization DTOs."

So today the pack is what causes those three containers to exist. Selecting the otel profile does not conjure services that no realization declares. Remove the nodes from the pack before APTL realizes them itself and evidence capture disappears rather than moves.

Symptom this also resolves

Grafana, Tempo and the collector currently run with container ports only and no host bindings, so nothing observability-related is reachable from the host:

aptl-grafana-otel     3000/tcp
aptl-tempo
aptl-otel-collector   4317-4318/tcp, 55678-55679/tcp

That contradicts ADR-012's http://localhost:3100, which the checked-in docker-compose.yml still publishes and which captured range snapshots under docs/raes/techvault-curated-live-validation-gate/ record as realized. Whether an operator-facing view is exposed, and on what port, becomes APTL's decision to make deliberately once it owns the stack. aptl lab start currently omits the Grafana line rather than printing a URL that answers nothing (#879), so nothing advertises a dead endpoint in the meantime.

Scope

• Realize the capture stack as backend infrastructure, independent of any pack declaring it.
• Tie it to the contract it serves: satisfy declared evidence_requirements, and be explicit about redaction, retention and loss disclosure rather than inheriting whatever the collector defaults to.
• Decide and document the operator-facing surface, including whether it is published and where, so ADR-012 and the realized range agree.
• Sequence with Remove the OTel nodes from TechVault: they are backend infrastructure, not scenario content OpenRAE/env-packs#337 so the pack's removal and APTL's ownership land together.

Found while making aptl lab start boot TechVault end to end (#879).

Requirements

• EXP-010 — Capture Plan Admission and Evidence Acquisition

Expanded scope — raes-env-packs 5.2.0

The user authorized expanding #992 on 2026-09-12 to implement all four evidence contracts in the published raes-env-packs 5.2.0 / raes 3.5.0 TechVault scenario. PR OpenRAE/env-packs#338 has merged; 5.2.0 removes the three observability nodes and the earlier cortex-job-index-schema-readback requirement.

The delivery must now support:

• cortex-enrichment-readback: exact enabled analyzer identity, bounded successful TechVaultScenarioContext_1_0 report for 172.20.1.30, and TheHive Cortex connector status, application/json, redact_secrets, checksum, run_lifetime, required loss disclosure.
• suricata-local-rule-readiness: admitted image/content identities and realized-byte digests, native configuration success, selected sources, all 16 loaded local SIDs, text/plain, redact_sensitive, checksum, run_lifetime, required loss disclosure; no rule bodies or host paths in evidence.
• suricata-login-sqli-alert: fresh participant-equivalent Kali POST /login containing UNION SELECT, exact Suricata SID 1000010 and Wazuh rule 303020 correlation, application/x-ndjson, redact_sensitive, checksum, run_lifetime, required loss disclosure.
• redteam-session-transcript: every interactive shell session on the red-team workstation, commands and responses attributable and ordered in time, full run from readiness through teardown, participant_output/text/plain, redact_secrets, chain_of_custody, run_lifetime, required loss disclosure. MCP calls alone are not coverage of every shell session.

Closed scopes prohibit intrusion. Open scopes require the least intrusive option that genuinely meets all evidence terms. Prefer existing native evidence over new agents; do not add the optional OTel stack when it would intrude unnecessarily. Any observability additions, including unasked or ephemeral ones, must be permitted by RAE scopes and reported as actual realized resources to the runtime. Required unsupported evidence rejects admission; this is not permission to claim the four contracts implemented by rejecting them forever. Retain all lifecycle, redaction, custody, loss, packaging, dependency compatibility, clean-machine lab, and workflow gates. Do not merge PRs without the user's explicit approval.

[Brad-Edwards]

🛠️ Picked up by /implement - driver codex, branch 992-own-observability-stack, 2026-09-11T00:48:18.862Z.

[Brad-Edwards]

gc workflow phase recorded: preflight (issue #992). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

[Brad-Edwards]

gc workflow phase recorded: preflight (issue #992). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

[Brad-Edwards]

gc workflow phase recorded: plan (issue #992). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

Implementation plan

Requirement: EXP-010. This also covers every #992 acceptance criterion and the user's open/closed-scope clarification. The existing EXP-010 experiment admission, collector, record, and archival contracts remain authoritative.

1. Backend ownership (TDD A): add one trusted backend-owned Collector/Tempo/Grafana Compose definition, composed on static, generated, mixed and image-free realization paths. Reuse Compose file assembly, model validation, host-port allocation, lifecycle ownership labels and asset packaging. Reject reserved service/container/network/volume ownership collisions before mutation. No synthetic scenario nodes or silent suppression of pack declarations.
2. Evidence admission (TDD A): connect typed SDL evidence intent to public experiment capture contracts and the existing registry. Start with the actual native service-index schema-readback source: require a fresh post-materialization projection, with source/scenario binding, bounded JSON, retained-byte checksum, redaction, run-scoped persistence and explicit loss outcome. Preserve the complete authored intent and its source/scope/window/trigger references; do not relabel api_response as a trace or treat a manifest union as proof. Exact supported policy terms are matched, not arbitrary strings accepted through boolean capabilities. Every unsupported required term rejects before provisioning; existing optional experiment degradation retains explicit limitation/comparability acceptance.
3. Scope and realized-world truth (TDD A): keep backend apparatus off scenario networks with only loopback operator/control-plane publications. Being backend-owned is not an exemption from RAES realization scopes. Any proposed in-world instrumentation must pass the public realization-envelope rules; closed-scope additions reject. Prefer existing non-invasive readback; do not install an agent merely to satisfy capture. Report all actual observability additions distinctly through public runtime observation/provenance and, where a run exists, augmentation disclosure, backed by native inventory evidence. Never claim authored addresses for added apparatus.
4. Evidence security/lifecycle defects (TDD B): fix malformed JSON/JSONL redaction escape and pre-limit chunk joins; redact Python/TypeScript telemetry at the shared serialization boundaries, remove Collector debug export, and preserve failure versus valid-empty/loss/finalization distinctions. Tempo is an operational buffer, not proof of durable run retention. Required capture may not rely on unknown SDK delivery or unauthenticated producer identity.
5. Operator and lifecycle integration (TDD A/B): reuse generated Grafana credentials with mandatory nonempty/non-placeholder validation; derive local endpoints from admitted port bindings and propagate them through existing producer launch seams. Verify actual Collector-to-Tempo readiness and bounded retrieval, normal finalization before teardown, prompt forced stop with disclosed interruption, and project-scoped cleanup even if the scenario bundle is gone. Remote backends without the required reachability/artifact guarantees reject the affected capture.
6. Compatibility and documentation (TDD A for executable assets; D for prose): remove obsolete in-tree scenario nodes and mappings and update the exact pack/serving-map compatibility set when the release containing Remove the OTel nodes from TechVault: they are backend infrastructure, not scenario content OpenRAE/env-packs#337 is available. The upstream cleanup PR SCN-010 inventory: reverse container (steady-state asset spec) #338 is currently open and held for this implementation: this is an explicit release dependency, not permission to merge upstream or to ship dual ownership. Preserve the existing dirty architecture edits, update ADR-012/operator documentation and EXP-010 traceability; no version/changelog edits.

Security passage: RAES parser/instantiation and envelope gates; contained/digest-checked bundle ingress; strict config/dotenv and managed-process env shape; trusted read-only Compose binds; loopback-only publication and private apparatus network; source/run identity and bounded HTTP; shared redaction before export and persistence; public evidence visibility filtering; create-once run storage and sealed archive ownership. Errors use existing bounded diagnostics, never source bodies/credentials/raw stderr.

Maintainability/extensibility: reuse DeploymentRealizationSpec and pure scenario lowering, compose assembly, host_ports, shared redactors, native schema verification, CaptureBinding + trusted source adapter, RunScope/ClockProvider, RunStorageBackend and terminal finalization. A future source or store extends those seams, not a second schema/registry/repository or pack-version branch.

Expected files: deployment Compose assembly/lifecycle helpers and a backend Compose asset; docker-compose.yml/config/otel and asset manifest; raes admission/provisioner/observation; capture registry/registrations and native-source adapter; evidence persistence/records/lifecycle; telemetry and existing endpoint launch integration; relevant pack pins/serving map/SDL fixtures; ADR-012/operator docs/EXP-010. Exact files follow the existing ownership boundaries.

Verification: observe failing tests before each executable change. Extend pytest coverage for capture admission, all required axes, native readback freshness, closed/open scope and complete disclosure, malformed/JSONL/oversized payloads, ownership collisions, generated/image-free paths, remapped ports/credentials, cleanup, and wheel assets. TypeScript changes get vitest tests; if aptl-mcp-common changes, rebuild and test every dependent MCP. Run the configured completion command and make policy, bounded code/test reviews, pre-commit run --all-files, strict docs build, CI and Sonar. Compose/config changes additionally require the fresh-machine clean stop -v/start lab gate, with actual ingestion/retrieval, operator login and teardown. No mocked/config-only result substitutes for that gate.

Delivery stops at reviewed PR readiness for the user's merge. An unavailable public contract or compatible pack release remains an explicit dependency requiring resolution, not a narrowed completion claim.

[Brad-Edwards]

gc workflow phase recorded: plan (issue #992). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

Implementation plan

Requirement: EXP-010. This also covers every #992 acceptance criterion and the user's open/closed-scope clarification. The existing EXP-010 experiment admission, collector, record, and archival contracts remain authoritative.

1. Backend ownership (TDD A): add one trusted backend-owned Collector/Tempo/Grafana Compose definition, composed on static, generated, mixed and image-free realization paths. Reuse Compose file assembly, model validation, host-port allocation, lifecycle ownership labels and asset packaging. Reject reserved service/container/network/volume ownership collisions before mutation. No synthetic scenario nodes or silent suppression of pack declarations.
2. Evidence admission (TDD A): connect typed SDL evidence intent to public experiment capture contracts and the existing registry. Start with the actual native service-index schema-readback source: require a fresh post-materialization projection, with source/scenario binding, bounded JSON, retained-byte checksum, redaction, run-scoped persistence and explicit loss outcome. Preserve the complete authored intent and its source/scope/window/trigger references; do not relabel api_response as a trace or treat a manifest union as proof. Exact supported policy terms are matched, not arbitrary strings accepted through boolean capabilities. Every unsupported required term rejects before provisioning; existing optional experiment degradation retains explicit limitation/comparability acceptance.
3. Scope and realized-world truth (TDD A): keep backend apparatus off scenario networks with only loopback operator/control-plane publications. Being backend-owned is not an exemption from RAES realization scopes. Any proposed in-world instrumentation must pass the public realization-envelope rules; closed-scope additions reject. Prefer existing non-invasive readback; do not install an agent merely to satisfy capture. Report all actual observability additions distinctly through public runtime observation/provenance and, where a run exists, augmentation disclosure, backed by native inventory evidence. Never claim authored addresses for added apparatus.
4. Evidence security/lifecycle defects (TDD B): fix malformed JSON/JSONL redaction escape and pre-limit chunk joins; redact Python/TypeScript telemetry at the shared serialization boundaries, remove Collector debug export, and preserve failure versus valid-empty/loss/finalization distinctions. Tempo is an operational buffer, not proof of durable run retention. Required capture may not rely on unknown SDK delivery or unauthenticated producer identity.
5. Operator and lifecycle integration (TDD A/B): reuse generated Grafana credentials with mandatory nonempty/non-placeholder validation; derive local endpoints from admitted port bindings and propagate them through existing producer launch seams. Verify actual Collector-to-Tempo readiness and bounded retrieval, normal finalization before teardown, prompt forced stop with disclosed interruption, and project-scoped cleanup even if the scenario bundle is gone. Remote backends without the required reachability/artifact guarantees reject the affected capture.
6. Compatibility and documentation (TDD A for executable assets; D for prose): remove obsolete in-tree scenario nodes and mappings and update the exact pack/serving-map compatibility set when the release containing Remove the OTel nodes from TechVault: they are backend infrastructure, not scenario content OpenRAE/env-packs#337 is available. The upstream cleanup PR SCN-010 inventory: reverse container (steady-state asset spec) #338 is currently open and held for this implementation: this is an explicit release dependency, not permission to merge upstream or to ship dual ownership. Preserve the existing dirty architecture edits, update ADR-012/operator documentation and EXP-010 traceability; no version/changelog edits.

Security passage: RAES parser/instantiation and envelope gates; contained/digest-checked bundle ingress; strict config/dotenv and managed-process env shape; trusted read-only Compose binds; loopback-only publication and private apparatus network; source/run identity and bounded HTTP; shared redaction before export and persistence; public evidence visibility filtering; create-once run storage and sealed archive ownership. Errors use existing bounded diagnostics, never source bodies/credentials/raw stderr.

Maintainability/extensibility: reuse DeploymentRealizationSpec and pure scenario lowering, compose assembly, host_ports, shared redactors, native schema verification, CaptureBinding + trusted source adapter, RunScope/ClockProvider, RunStorageBackend and terminal finalization. A future source or store extends those seams, not a second schema/registry/repository or pack-version branch.

Expected files: deployment Compose assembly/lifecycle helpers and a backend Compose asset; docker-compose.yml/config/otel and asset manifest; raes admission/provisioner/observation; capture registry/registrations and native-source adapter; evidence persistence/records/lifecycle; telemetry and existing endpoint launch integration; relevant pack pins/serving map/SDL fixtures; ADR-012/operator docs/EXP-010. Exact files follow the existing ownership boundaries.

Verification: observe failing tests before each executable change. Extend pytest coverage for capture admission, all required axes, native readback freshness, closed/open scope and complete disclosure, malformed/JSONL/oversized payloads, ownership collisions, generated/image-free paths, remapped ports/credentials, cleanup, and wheel assets. TypeScript changes get vitest tests; if aptl-mcp-common changes, rebuild and test every dependent MCP. Run the configured completion command and make policy, bounded code/test reviews, pre-commit run --all-files, strict docs build, CI and Sonar. Compose/config changes additionally require the fresh-machine clean stop -v/start lab gate, with actual ingestion/retrieval, operator login and teardown. No mocked/config-only result substitutes for that gate.

Delivery stops at reviewed PR readiness for the user's merge. An unavailable public contract or compatible pack release remains an explicit dependency requiring resolution, not a narrowed completion claim.

Scope-preserving realization refinement

The TechVault Docker authority is explicitly host-root-equivalent and uses the selected daemon socket. Therefore a private network alone does not make same-daemon apparatus outside its in-principle visible world. The legacy/unspecified RAE realization posture resolves closed. The implementation must use RAE's public lexical designation resolver, not a backend-owned exception. For the supported native-readback requirement, the evidence does not depend on OTel: the scope-preserving default is to omit optional visible apparatus when additions are forbidden and retain the required native evidence. Any requirement that actually needs prohibited instrumentation remains an admission rejection. Explicitly open applicable scopes may admit the full added footprint, which must be inventoried and disclosed through public runtime/run carriers. The operator-view limitation must be reported, not silently presented as a running Grafana. A separately isolated observability host is an alternative architecture, not assumed authority for this issue.

Discovered security repair

Evidence persistence retained malformed structured payloads, and both exporter boundaries could emit sensitive attributes/events/resources. Regression tests precede their fixes. The required common-MCP install also reported vulnerable transitive dependencies and a test-tool advisory; compatible lockfile updates plus the patched 4.1.11 test toolchain are included and will be verified with the dependent MCP builds/tests.

[Brad-Edwards]

Implementation pause: hard external dependency / coordinated release decision. Rechecked OpenRAE/env-packs PR #338: OPEN, unmerged, explicitly held until aptl#992 lands. Latest release remains v5.1.0 and does not contain that cleanup. The ownership cutover cannot accept the existing pack declarations without conflicting ownership; no compatibility shim, silent node suppression, upstream merge, or release has been performed. User direction is required to break the release sequencing loop.

The local branch 992-own-observability-stack contains uncommitted partial implementation: separate backend Compose asset, exact required-evidence policy admission and native readback acquisition, public open/closed designation-based optional-stack selection, collision rejection, and shared export/persistence redaction repairs. Latest targeted verification: 80 passed across test_raes_evidence.py, test_backend_observability.py, test_evidence_security.py, and test_telemetry.py. The common MCP library previously passed its build and 463 tests with a clean dependency audit.

Open execution obligations remain: complete native inventory and runtime/augmentation disclosure for every added observability resource, listener-observer scope enforcement, endpoint/lifecycle/readiness integration, compatible pack and serving-map cutover, dependent MCP verification, full completion/policy/pre-commit gates, clean-machine live lab validation, traceability, bounded reviews, and publication/CI/readiness. No completion or merge-readiness claim is made. Resume these obligations after the coordinated release decision; preserve the existing working tree and architecture edits.

[Brad-Edwards]

gc workflow phase recorded: plan (issue #992). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

Implementation plan

Requirement: EXP-010. This also covers every #992 acceptance criterion and the user's open/closed-scope clarification. The existing EXP-010 experiment admission, collector, record, and archival contracts remain authoritative.

1. Backend ownership (TDD A): add one trusted backend-owned Collector/Tempo/Grafana Compose definition, composed on static, generated, mixed and image-free realization paths. Reuse Compose file assembly, model validation, host-port allocation, lifecycle ownership labels and asset packaging. Reject reserved service/container/network/volume ownership collisions before mutation. No synthetic scenario nodes or silent suppression of pack declarations.
2. Evidence admission (TDD A): connect typed SDL evidence intent to public experiment capture contracts and the existing registry. Start with the actual native service-index schema-readback source: require a fresh post-materialization projection, with source/scenario binding, bounded JSON, retained-byte checksum, redaction, run-scoped persistence and explicit loss outcome. Preserve the complete authored intent and its source/scope/window/trigger references; do not relabel api_response as a trace or treat a manifest union as proof. Exact supported policy terms are matched, not arbitrary strings accepted through boolean capabilities. Every unsupported required term rejects before provisioning; existing optional experiment degradation retains explicit limitation/comparability acceptance.
3. Scope and realized-world truth (TDD A): keep backend apparatus off scenario networks with only loopback operator/control-plane publications. Being backend-owned is not an exemption from RAES realization scopes. Any proposed in-world instrumentation must pass the public realization-envelope rules; closed-scope additions reject. Prefer existing non-invasive readback; do not install an agent merely to satisfy capture. Report all actual observability additions distinctly through public runtime observation/provenance and, where a run exists, augmentation disclosure, backed by native inventory evidence. Never claim authored addresses for added apparatus.
4. Evidence security/lifecycle defects (TDD B): fix malformed JSON/JSONL redaction escape and pre-limit chunk joins; redact Python/TypeScript telemetry at the shared serialization boundaries, remove Collector debug export, and preserve failure versus valid-empty/loss/finalization distinctions. Tempo is an operational buffer, not proof of durable run retention. Required capture may not rely on unknown SDK delivery or unauthenticated producer identity.
5. Operator and lifecycle integration (TDD A/B): reuse generated Grafana credentials with mandatory nonempty/non-placeholder validation; derive local endpoints from admitted port bindings and propagate them through existing producer launch seams. Verify actual Collector-to-Tempo readiness and bounded retrieval, normal finalization before teardown, prompt forced stop with disclosed interruption, and project-scoped cleanup even if the scenario bundle is gone. Remote backends without the required reachability/artifact guarantees reject the affected capture.
6. Compatibility and documentation (TDD A for executable assets; D for prose): remove obsolete in-tree scenario nodes and mappings and update the exact pack/serving-map compatibility set when the release containing Remove the OTel nodes from TechVault: they are backend infrastructure, not scenario content OpenRAE/env-packs#337 is available. The upstream cleanup PR SCN-010 inventory: reverse container (steady-state asset spec) #338 is currently open and held for this implementation: this is an explicit release dependency, not permission to merge upstream or to ship dual ownership. Preserve the existing dirty architecture edits, update ADR-012/operator documentation and EXP-010 traceability; no version/changelog edits.

Security passage: RAES parser/instantiation and envelope gates; contained/digest-checked bundle ingress; strict config/dotenv and managed-process env shape; trusted read-only Compose binds; loopback-only publication and private apparatus network; source/run identity and bounded HTTP; shared redaction before export and persistence; public evidence visibility filtering; create-once run storage and sealed archive ownership. Errors use existing bounded diagnostics, never source bodies/credentials/raw stderr.

Maintainability/extensibility: reuse DeploymentRealizationSpec and pure scenario lowering, compose assembly, host_ports, shared redactors, native schema verification, CaptureBinding + trusted source adapter, RunScope/ClockProvider, RunStorageBackend and terminal finalization. A future source or store extends those seams, not a second schema/registry/repository or pack-version branch.

Expected files: deployment Compose assembly/lifecycle helpers and a backend Compose asset; docker-compose.yml/config/otel and asset manifest; raes admission/provisioner/observation; capture registry/registrations and native-source adapter; evidence persistence/records/lifecycle; telemetry and existing endpoint launch integration; relevant pack pins/serving map/SDL fixtures; ADR-012/operator docs/EXP-010. Exact files follow the existing ownership boundaries.

Verification: observe failing tests before each executable change. Extend pytest coverage for capture admission, all required axes, native readback freshness, closed/open scope and complete disclosure, malformed/JSONL/oversized payloads, ownership collisions, generated/image-free paths, remapped ports/credentials, cleanup, and wheel assets. TypeScript changes get vitest tests; if aptl-mcp-common changes, rebuild and test every dependent MCP. Run the configured completion command and make policy, bounded code/test reviews, pre-commit run --all-files, strict docs build, CI and Sonar. Compose/config changes additionally require the fresh-machine clean stop -v/start lab gate, with actual ingestion/retrieval, operator login and teardown. No mocked/config-only result substitutes for that gate.

Delivery stops at reviewed PR readiness for the user's merge. An unavailable public contract or compatible pack release remains an explicit dependency requiring resolution, not a narrowed completion claim.

User clarification: minimum intrusion in every open scope

This supersedes any earlier wording that enabled visible apparatus merely because the governing scopes were open. Closed scopes prohibit intrusion; open scopes permit only the least intrusive supported realization that genuinely meets the need. First require full evidence-contract coverage, then choose the least intrusive compliant option. Existing native readback wins over added agents or the OTel stack when it satisfies all SDL evidence, even when every relevant scope is open. No evidence obligation may be weakened to avoid intrusion. Every actual observability addition remains a runtime-disclosure obligation.

TDD A refinement: update raes_observability_scope.py selection so the presently supported native-readback capture keeps same-daemon visible optional apparatus omitted for both open and closed scenarios. Preserve public lexical scope resolution and private apparatus that does not intrude into the scenario world. Extend tests/test_raes_evidence.py to exercise open/closed selection, retained native evidence without the stack, and rejection of unsupported intent regardless of openness. Update ADR-012 and the preflight note so always-on activation cannot override scope or minimum-intrusion selection. Future intrusive collector support must prove necessity and compare supported candidates before selecting a permitted addition; this change does not invent support for such collectors.

The separate upstream release-sequencing pause remains unresolved; this clarification authorizes neither an upstream merge nor a compatibility bypass. Complete the local clarification and its targeted tests while preserving all previously recorded implementation and verification obligations.

[Brad-Edwards]

Minimum-intrusion clarification applied locally. Open scopes no longer automatically select the same-daemon visible observability stack: the currently supported native readback meets the evidence need without it, so the stack is omitted in open as well as closed scopes. Private apparatus that does not intrude into the scenario world remains a separate case. Closed scope is a prohibition, not a cost to trade against evidence coverage. Future intrusive source support must establish necessity, choose the least intrusive compliant option, and disclose every actual addition.

TDD evidence: three new/updated assertions failed against the previous open-scope selection, then the focused evidence/observability suite passed (69 tests). The nine scope/readback cases passed again after formatting. Scoped Ruff lint/format checks and git diff --check pass. The evidence regressions verify required native records remain retained without the stack, stale readback fails, and unsupported evidence terms reject for both scope postures. ADR-012 and the architecture preflight now supersede unconditional always-on activation with the minimum-intrusion rule.

This is not completion or readiness. Changes remain uncommitted; the previously recorded disclosure, lifecycle, compatibility, traceability, review, and full verification obligations remain open. Rechecked env-packs PR #338: still OPEN and unmerged, latest release still v5.1.0. The separate coordinated-release decision is unresolved; the user clarification did not authorize an upstream merge or release.

[Brad-Edwards]

Release dependency resolved: verified the published raes-env-packs 5.2.0 wheel from PyPI against SHA-256 ee90cb12bc191f4d449f86c666492bab08483b92690bcee1d9d24377fab28f6e. PR OpenRAE/env-packs#338 is merged (7c01acd0ec3b889ca015178c886af3bede95d796). The wheel contains no Collector, Tempo, or Grafana scenario nodes. Its declared dependency is raes==3.5.0; the complete published TechVault SDL parses with that release in an isolated environment (34 nodes, root realization posture open).

The published evidence contract is materially different from the pack used for the current plan and TDD. The cortex-job-index-schema-readback requirement is gone. Four requirements are now present: cortex-enrichment-readback, suricata-local-rule-readiness, suricata-login-sqli-alert, and redteam-session-transcript. The last requires every interactive red-team shell session throughout range readiness to teardown, commands and responses ordered and attributable, text/plain with redaction, run-lifetime retention, loss disclosure, and chain_of_custody. Current collector registrations explicitly declare no chain-of-custody support; the finite native schema-readback adapter does not cover these four contracts. Ordinary MCP tool traces are not proof of every interactive shell session.

The old release-wait pause is lifted. A separate material scope/security-design decision is now requested: whether #992 should include implementing the four new evidence contracts, especially full-session chain of custody, before the pack cutover. No dependency pins, lockfiles, or serving-map identity were changed by this inspection. Until coverage is genuinely implemented, the contract requires admission rejection, not weakened evidence promises. Minimum intrusion, closed-scope prohibition, truthful disclosure of all actual additions, and every previously recorded implementation/verification obligation remain in force.

[Brad-Edwards]

Scope-expansion decision: the user explicitly instructed "Expand scope" after the 5.2.0 evidence-contract assessment. Resume #992 and include all four published contracts: Cortex enrichment readback, Suricata local-rule readiness, exact Suricata/Wazuh alert correlation, and complete red-team interactive session transcripts with chain of custody. The old release wait and the subsequent expansion-decision pause are both resolved. This authorization does not waive any evidence term, permit intrusion into closed scope, permit unnecessary intrusion in open scope, or authorize PR merges.

Refresh the architecture preflight against the published 5.2.0 wheel and required RAE 3.5.0 before completing the revised plan. Preserve all existing uncommitted implementation and architecture work. In particular, investigate the public custody contract and actual session ingress/PTY coverage rather than treating CLI or MCP spans as every shell session. Use existing native readback/log surfaces where sufficient; every necessary added observer needs public scope admission, native realized-world disclosure, bounded loss accounting and retained run evidence. Same-daemon Docker host-root authority must be considered when claiming an apparatus or custody store is isolated. The verified wheel is locally available at /tmp/aptl-992-pack-W9BJxe/raes_env_packs-5.2.0-py3-none-any.whl for inspection. No repository dependency pins have yet been changed.

[Brad-Edwards]

gc workflow phase recorded: preflight (issue #992). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

[Brad-Edwards]

gc workflow phase recorded: preflight (issue #992). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.