Cover the known realization failures in the existing boot test

[Brad-Edwards]

The small boot test missed real regressions because its scenario did not exercise affected backend behavior. Extend #985’s owned fixture with the critical cases that exposed those defects.

Done when

• Choose a small set of regressions covering required content execution, service setup, port readback and any selected orchestration path.
• Each claimed check observes an actual effect; unit tests cover cases too expensive for every boot.
• Keep a short mapping from advertised realization concerns to existing tests and explicitly untested areas.
• The installed-wheel job still finishes in a practical budget and verifies cleanup.

Automatic envelope coverage is deferred to #1126; shared contract tests remain in #1099. Do not make one giant boot scenario or duplicate the existing CI job.

Related: #985, #969.

The original automatic mapping against every advertised realization concern is retained in #1126. #1099 owns shared contract tests; it does not replace that coverage requirement.

[Brad-Edwards]

🛠️ Picked up by /implement - driver claude, branch 993-cover-boot-test-regressions, 2026-09-20T21:33:56.020Z.

[Brad-Edwards]

gc workflow phase recorded: preflight (issue #993). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

[Brad-Edwards]

gc workflow phase recorded: plan (issue #993). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

Plan — cover the known realization failures in the existing boot test

Requirement-free run. The architecture preflight note
(docs/architecture/issue-993-boot-realization-regressions-preflight.md) is
binding; this plan implements it. One fixture, one clean-install-lab-boot
job, no second qualification system.

What is missing today

tests/fixtures/materialization-envelope.sdl.yaml declares packages, local
identity and a filesystem entry. The boot gate therefore never exercises
content placement, service-manager units, listener/port realization, or the
orchestrator adapter — the four surfaces the issue names. Every one of them has
shipped a defect that the boot gate could not have caught.

The regression chain (AC1, AC2)

Extend the one shared fixture with a single causal chain on the existing
smoke-box node, so no assertion can pass vacuously:

1. Required content execution — inline, non-sensitive content writes
   /etc/ssh/sshd_config.d/10-aptl-smoke.conf containing Port 2022.
2. Service setup — runtime.packages adds openssh-server (apt);
   runtime.service_manager_units declares ssh.service
enabled_state: enabled, active_state: active. This selects the
   init-capable aptl/generic-systemd-base-debian substrate.
3. Port readback — runtime.service_listeners declares tcp/2022;
   runtime.network.published_ports declares container 2022 on an explicit
   127.0.0.1:32022 host binding.
4. Orchestration — one end-only RAES workflows resource traverses the
   ordinary AptlOrchestrator/WorkflowEngine adapter and persists a terminal
   result plus history under runs/<run>/orchestration/<address>/.

The content is causal: without it sshd binds 22, so the listener wait, the port
tuple and the connect probe all fail. Debian 13 ships
Include /etc/ssh/sshd_config.d/*.conf at sshd_config:12, above #Port 22,
and presets ssh.service (not ssh.socket), so the authored port wins.
Verified locally against the real containers/generic-systemd-base-debian
image with APTL's own init run flags: UnitFileState=enabled,
ActiveState=active, SubState=running, Result=success,
docker port → 2022/tcp -> 127.0.0.1:32022, and a host connect returning
SSH-2.0-OpenSSH_10.0. No credential, no interactive login, no public bind.

runtime.orchestration_authorities is explicitly out: it is a
host-root-equivalent Docker authority surface (ADR-055, #949, #974), not the
ordinary workflow adapter.

How the effects are observed (AC2)

New bounded verifier scripts/ci/assert_boot_realization.py, in the style of
the existing scripts/ci/assert_project_teardown.py and run from the same
installed venv. It is parameterized by project dir, node address, expected
(host_ip, host_port, container_port, protocol) tuple and workflow address —
safe identities, not a new schema. It reuses the installed package's own
identities and rules rather than restating them:

• resolve exactly one container via WorkspaceOwnership + the Compose project
  label + aptl.node.address (never a bare name or prefix);
• read the placed file's exact bytes inside that container and compare to
  the authored text (the backend's own observe_file readback is presence-only,
  so the gate checks content itself);
• systemctl show the unit and require UnitFileState=enabled,
  ActiveState=active, Result=success;
• observe the live listener through the installed
  _proc_net_listeners reader — from outside the container's trust boundary,
  so a shadowed ss cannot change the answer;
• docker inspect the port map and require the exact loopback tuple, then
  make one bounded argv-based TCP connection to it. A wildcard bind fails;
• discover the single run directory and parse the one expected workflow with
  RAES WorkflowExecutionState/WorkflowResultContract, requiring the terminal
  SUCCEEDED state and a non-empty history. PENDING, a filename, or unparsed
  JSON is not acceptance.

Failures print safe identifiers and expected/actual non-secret state only — no
inspect payloads, config bodies, environment or tracebacks. Content never
travels through argv.

Port-realization defect found while mapping this surface (AC1, AC2)

_append_base_ports substitutes the container port for an absent host_port,
so an image-free node declaring {container_port: 2022} with no host port gets
an exact -p 127.0.0.1:2022:2022/tcp binding — while the Compose path asks
Docker for an ephemeral publish for the identical declaration, and
published_port_conflicts deliberately skips the probe because "a binding with
no host port cannot conflict". A busy host port then fails lab start with a raw
Docker error instead of APTL's fail-closed message. Fix, authorized in-thread:
emit an empty host port so Docker performs the ephemeral publish both paths
already document. tests/test_compose_base_substrate.py::test_published_port_defaults_host_port_to_container_port
pins the old behavior and is rewritten to the corrected contract, with a
regression test asserting the two paths agree.

Focused tests for what is too expensive to boot (AC2)

• tests/test_boot_realization_gate.py (new): drive
  assert_boot_realization.py against recorded/faked observations — missing
  content, a disabled or inactive unit, a failed unit result, an absent
  listener, a wildcard or mismatched port tuple, a refused connection, a
  PENDING workflow, a missing history, a missing or ambiguous container —
  each must fail closed with a bounded message and a non-zero exit. No Docker.
• tests/test_compose_base_substrate.py: the corrected ephemeral publish and
  the explicit-binding case.
• tests/test_platform_ci_gates.py: structural contracts for the extended job —
  it uses the shared fixture, invokes the verifier with the tuple the SDL
  actually declares (parsed from the fixture, so the test cannot pass while
  the field is absent), keeps stop and the cleanup proof under always(),
  and retains its timeout.
• tests/test_scenarios.py, tests/test_lab_fresh_start.py,
  tests/test_imagefree_admission_integration.py: keep the shared-fixture
  consumers coherent with the richer SDL.
  test_admit_and_realize_service_node_boots_a_real_service is narrowed
  rather than deleted — it exercises the dnf/Rocky systemd substrate, which the
  apt/Debian boot chain does not cover; its docstring states that reduced claim.

Coverage map (AC3)

New short doc docs/testing/boot-realization-coverage.md, linked from
docs/testing/platform-pr-gates.md and the docs nav: one table mapping each
concern the realization envelope advertises
(src/aptl/backends/raes_realization_envelope.py) to the tests that cover it
and to the areas explicitly not covered by the small boot
(architecture, image, resource-allocation, account-placement, acl,
generated artifacts, persistent volumes, forwarding agents, orchestration
authorities). Human-maintained and deliberately short; automatic derivation
from the envelope is #1126's and must not be started here.
docs/testing/platform-pr-gates.md is updated to describe what the job now
proves.

Budget and cleanup (AC4)

The chain adds one systemd base-image build (cached layers), one
apt-get install openssh-server, one unit start and four readbacks to a job
with a 60-minute timeout that today runs well inside it. I will measure the
added wall-clock on the local full-CLI run and record it in the PR; if the
margin is not comfortable the listener wait is bounded and the chain shrinks to
the service/port pair. stop --volumes --yes and
assert_project_teardown.py stay always() and unchanged, so cleanup
verification is untouched.

Verification

Targeted pytest for the touched modules, pre-commit run --hook-stage pre-commit,
mkdocs build --strict and Vale for the docs, plus — per the repo
rule that the lab is the product — a real
aptl lab init → aptl lab start → verifier → aptl lab stop --volumes
→ teardown proof cycle on this box from a fresh init directory using the
built wheel, not the checkout. CI owns the repository-wide suites.

Not in scope

TechVault qualification, runtime Docker authority, spawned workloads,
participants, generated credentials, remote Docker, non-Linux hosts, a second
boot job or SDL, automatic envelope coverage (#1126), shared contract tests
(#1099), and any RAES/backend/schema redesign. No CHANGELOG.md or version
edit; the PR title carries the release semantics.

[Brad-Edwards]

gc_codex_review — sanitized deferred publication for issue #993, cycle 1 of 1
Reviewed revision: 775f2f97bd29547b930bee5204624b893845d3d5d50d4ff8bb4a1f8009a71985

Architectural read

Core reviewer: The change is shaped correctly around one shared scenario and the existing installed-wheel gate. It reuses backend ownership and observation interfaces, RAES workflow contracts, and the run-store persistence boundary. The history batching and ephemeral-port fixes match their existing contracts. The main gap is in the verifier: a helper intended for host-port occupancy drops information required to prove an exact publication.

Security reviewer: This change extends the existing installed-wheel boot gate through the shared scenario and a parameterized verifier, alongside focused fixes to workflow-history persistence and omitted host-port handling. It reuses backend ownership enforcement and run-store redaction, while explicitly publishing the fixture's service on loopback. The fixture/verifier seam supports further regression checks without introducing another execution architecture. I found no concrete exploitable security regression introduced by the supplied diff.

Verdict: ship-with-fixes

Findings

core-F1 — Preserve the destination container port when verifying publications

• Classification: one-off
• Decision: fix
• Rationale: Real. owned_bindings() returns only (host_ip, host_port, protocol) because its existing caller uses it for host-port occupancy, so the boot gate's exact-publication check cannot distinguish 127.0.0.1:32022 targeting container port 22 from the declared 2022. The listener check proves 2022 is bound and the greeting proves an SSH daemon answered, but neither proves which container port the host binding targets, so a mismapped publication passes while sshd listens on both. Fixing by parsing the full host-IP/host-port/container-port/protocol tuple inside the gate rather than widening the shared helper, keeping the extra-binding rejection, with a regression that drives wrong-target inspect data through both the observation and the assertion.

[Brad-Edwards]

gc_codex_review pre-push cycle 1 of 1 complete for issue #993 on branch '993-cover-boot-test-regressions'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next gc_codex_review (uncommitted) invocation to count cycles.

[Brad-Edwards]

Review decision record — codex cycle 1 (issue #993)

Reviewer: codex
Cycle: 1
Verdict: ship-with-fixes

Architectural read:

Core reviewer: The change is shaped correctly around one shared scenario and the existing installed-wheel gate. It reuses backend ownership and observation interfaces, RAES workflow contracts, and the run-store persistence boundary. The history batching and ephemeral-port fixes match their existing contracts. The main gap is in the verifier: a helper intended for host-port occupancy drops information required to prove an exact publication.

Security reviewer: This change extends the existing installed-wheel boot gate through the shared scenario and a parameterized verifier, alongside focused fixes to workflow-history persistence and omitted host-port handling. It reuses backend ownership enforcement and run-store redaction, while explicitly publishing the fixture's service on loopback. The fixture/verifier seam supports further regression checks without introducing another execution architecture. I found no concrete exploitable security regression introduced by the supplied diff.

Blocking findings: 1

Finding 1 — one-off

• ID: core-F1
• Title: Preserve the destination container port when verifying publications
• Decision: fix
• Rationale: Real. owned_bindings() returns only (host_ip, host_port, protocol) because its existing caller uses it for host-port occupancy, so the boot gate's exact-publication check cannot distinguish 127.0.0.1:32022 targeting container port 22 from the declared 2022. The listener check proves 2022 is bound and the greeting proves an SSH daemon answered, but neither proves which container port the host binding targets, so a mismapped publication passes while sshd listens on both. Fixing by parsing the full host-IP/host-port/container-port/protocol tuple inside the gate rather than widening the shared helper, keeping the extra-binding rejection, with a regression that drives wrong-target inspect data through both the observation and the assertion.

[Brad-Edwards]

Pre-PR base synchronization

• Source: refs/remotes/origin/dev at 3ce89ad5d65251d646b78589e5d9a82a420a7132
• Outcome: already_current
• Published feature head: 5b4d9018412bcb62e582fb1d821b06fc59b906e9
• Synchronized tree: 6f24b311d781e28eb757db6ba2ddec57f661f7d2

[Brad-Edwards]

Pre-PR base synchronization

• Source: refs/remotes/origin/dev at 3ce89ad5d65251d646b78589e5d9a82a420a7132
• Outcome: already_current
• Published feature head: 8ca805f1a1f4236fd41001ac0ab8f5a2d80bb754
• Synchronized tree: 6b267c0fc16552dfcddb7e6aaeb48b5763d4e4fb

[Brad-Edwards]

Ready for review — issue #993

PR: #1142
Plan: #993 (comment)

Outcome

The boot gate now proves the range was really built, not just that the CLI exited zero: one causal chain means a config file that never lands takes the service, the port and the connection down with it. Building it exposed two shipped defects, both fixed here, so a bad build fails CI in about eighty seconds instead of reaching a laptop, and the run archive keeps a real record of what the orchestrator did.

Files changed

Added:

• scripts/ci/assert_boot_realization.py
• tests/test_boot_realization_gate.py
• docs/testing/boot-realization-coverage.md
• docs/architecture/issue-993-boot-realization-regressions-preflight.md

Modified:

• .github/workflows/checks.yml
• src/aptl/core/deployment/_compose_base_substrate.py
• src/aptl/backends/raes_orchestrator.py
• tests/fixtures/materialization-envelope.sdl.yaml
• tests/test_compose_base_substrate.py
• tests/test_raes_orchestrator.py
• tests/test_platform_ci_gates.py
• tests/test_imagefree_admission_integration.py
• docs/testing/platform-pr-gates.md
• docs/architecture/index.md
• mkdocs.yml
• docs/reviews/962-lilrae-readiness/tracked-file-inventory.tsv

Reviews

• codex: ship-with-fixes, 1 cycle. One finding: the gate's publication check dropped the destination container port, so a 32022-to-22 mismapping observed as correct. Fixed by parsing the full tuple, with regressions, re-proven live.
• codex-security: No exploitable regression. Reuses backend ownership and run-store redaction, publishes on loopback only, adds no credential, Docker socket or privileged flag, keeps content out of argv and gate output.

Traceability reconciliation

Proposed in this PR — verified against the merged tree in Phase E.

• IMPLEMENTS / TESTS / DOCUMENTS added: 0
• Links updated: 0
• Stale links removed: 0

Status

• CI: ✅ green
• SonarCloud: ✅ passed
• PR ready for user review and merge. Phase E validates the merged requirement state, posts the final report, and closes the issue automatically once this PR merges.

[Brad-Edwards]

Delivery readiness recorded — Phase E finalizes issue #993 automatically when PR #1142 merges.

[Brad-Edwards]

Final report — issue #993 complete

PR: #1142
Plan: #993 (comment)

Outcome

The boot gate now proves the range was really built, not just that the CLI exited zero: one causal chain means a config file that never lands takes the service, the port and the connection down with it. Building it exposed two shipped defects, both fixed here, so a bad build fails CI in about eighty seconds instead of reaching a laptop, and the run archive keeps a real record of what the orchestrator did.

Files changed

Added:

• scripts/ci/assert_boot_realization.py
• tests/test_boot_realization_gate.py
• docs/testing/boot-realization-coverage.md
• docs/architecture/issue-993-boot-realization-regressions-preflight.md

Modified:

• .github/workflows/checks.yml
• src/aptl/core/deployment/_compose_base_substrate.py
• src/aptl/backends/raes_orchestrator.py
• tests/fixtures/materialization-envelope.sdl.yaml
• tests/test_compose_base_substrate.py
• tests/test_raes_orchestrator.py
• tests/test_platform_ci_gates.py
• tests/test_imagefree_admission_integration.py
• docs/testing/platform-pr-gates.md
• docs/architecture/index.md
• mkdocs.yml
• docs/reviews/962-lilrae-readiness/tracked-file-inventory.tsv

Reviews

• codex: ship-with-fixes, 1 cycle. One finding: the gate's publication check dropped the destination container port, so a 32022-to-22 mismapping observed as correct. Fixed by parsing the full tuple, with regressions, re-proven live.
• codex-security: No exploitable regression. Reuses backend ownership and run-store redaction, publishes on loopback only, adds no credential, Docker socket or privileged flag, keeps content out of argv and gate output.

Traceability reconciliation

• IMPLEMENTS / TESTS / DOCUMENTS added: 0
• Links updated: 0
• Stale links removed: 0

Requirement-free run: the issue declares no UIDs and adds no new structural gate needing an anchor. It extends an existing gate DEP-008 and SEC-008 already trace; #1126 keeps automatic envelope coverage and #1099 the shared contract tests. Follow-up PR #1144 retired the redundant dnf live scenario per the preflight and recorded the resulting gap.

Status

• CI: ✅ green
• SonarCloud: ✅ passed
• PR ready for user review and merge.