Investigate transient Wazuh API TLS handshake retries during clean boot

[Brad-Edwards]

Problem

A clean aptl lab start can emit repeated authenticated-readiness probe failures while the Wazuh API is coming up:

curl_safe: curl exit 35 for https://localhost:55000/security/user/authenticate

The retry loop ultimately recovers and startup completes successfully, but the diagnostic is noisy and may hide a timing, certificate reload, or readiness-ordering problem.

Reproduction

Observed on the release candidate at commit 2794ab930450a4e59ca31d2a2167f7bee33101c1 in both required clean-install paths:

• candidate wheel install, startup run run_20260911T235830Z;
• fresh source checkout with editable install, startup run run_20260912T003637Z.

In both cases, the command later advanced through Wazuh readiness, SSH validation, MCP build, SOC seeding, terminal snapshot capture, and exited zero with Lab is ready. The final inventory contained 32 running project containers.

Investigation goals

• Identify whether exit 35 is caused by Wazuh API TLS initialization, certificate availability/reload, protocol negotiation, or probe ordering.
• Determine whether the readiness probe should wait on a stronger prerequisite before attempting authentication.
• Preserve bounded retries and fail-closed terminal behavior.
• Add a regression test or deterministic diagnostic that distinguishes an expected warm-up condition from a persistent TLS failure.

Acceptance

• Clean boot does not emit avoidable TLS handshake failures during normal Wazuh warm-up, or the warm-up state is represented by a clear, non-error readiness message.
• A persistent TLS/authentication failure still causes startup to fail with an actionable diagnostic.
• Relevant clean-boot and readiness tests pass on Linux and macOS runners.

[Brad-Edwards]

Reproduced again during final manual release QA for candidate d2555cb0985f5ca4ce648eb149f0cb3526c729a4:

• Candidate-wheel path (run_20260912T040848Z): three curl exit 35 retries against the Wazuh authentication endpoint, then recovery.
• Exact-source path (run_20260912T043941Z): four retries, then recovery.
• Both clean boots reached Lab is ready; all downstream manual QA rows passed (18/18 per path), and both scoped teardowns passed.

This remains recoverable/non-blocking for the release candidate, but it is reproducible on both supported install paths.

[Brad-Edwards]

🛠️ Picked up by /implement - driver claude-code, branch 1002-wazuh-api-tls-warmup, 2026-09-14T04:39:51.201Z.

[Brad-Edwards]

gc workflow phase recorded: preflight (issue #1002). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

[Brad-Edwards]

gc workflow phase recorded: plan (issue #1002). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

Plan: quiet, phase-classified Wazuh API readiness (#1002)

Root cause (observed, not inferred)

• The noise comes from the RAES backend's authenticated-readiness loop (_compose_stateful_readiness.py), which runs during "Starting containers", not the later "Waiting for Wazuh services" step. In the QA transcript, all four curl exit 35 lines precede that step.
• The generated env-pack Compose (.aptl/realization/compose-base.yml) gives aptl-wazuh-manager no healthcheck. wait_for_realized_health() therefore treats a merely running manager as settled, and the authenticated probe starts before the API listens. The status output confirms this: aptl-wazuh-manager: running, with no (healthy).
• Exit 35 is Docker's published-port proxy, not a TLS or certificate fault. Local experiment on Docker 29.5.0: with a container publishing 55000 and nothing listening, curl -sfk https://localhost:<port> returns 35. An unpublished port returns 7.
• Standalone wazuh/wazuh-manager:4.12.0 timeline, sampled every second: ~15s of exit 35, then ~4s of HTTP 401 on the root path while authentication is still rejected (API user not yet provisioned), then authentication succeeds. Those ~15–20s at a 5s interval match the 3–4 reported retries.
• curl_json() logs every non-zero exit as a WARNING, so each expected warm-up poll prints a TLS-looking error.
• A regression test at the real subprocess boundary is already red on dev for exactly this log line.

Design (per preflight note docs/architecture/issue-1002-wazuh-api-tls-warmup-preflight.md)

1. aptl.utils.curl_safe: add curl_request(), which returns a secret-free CurlOutcome: numeric curl exit, HTTP status (or None for no HTTP response), parsed JSON payload (repr=False), and a normalized transport category (tls_handshake, connection_refused, timeout, …) derived from the exit code only, never from stderr.

   • It never logs at WARNING. The readiness owner decides severity.
   • curl_json() and curl_status() keep their contracts and warning behavior, but share the TLS-argument and 0600 header/body temp-file helpers, so secret handling stays in one place.

2. aptl.core.services: probe_manager_api() runs three phases per attempt and returns a bounded ManagerApiProbe (phase, category, curl exit, HTTP status, no secrets):

   • transport: credential-free GET /. Any HTTP status proves TLS and HTTP completed.
   • authentication: the existing Basic-auth request and token shape check.
   • manager_status: the existing affected_items check.

   check_manager_api_ready() becomes probe(...).ready. Warm-up states log at DEBUG only. No retries inside the probe.

3. _compose_stateful_readiness: keeps the single ReadinessPolling deadline and interval.

   • It keeps the last classified observation per service. At the deadline, LabResult.error names the service, phase, category, curl exit or HTTP code, and budget, plus a container-logs action.
   • This is still fail-closed. authenticated_readiness stays the boolean evidence map.
   • The indexer uses the existing check_indexer_status() classification.
   • Fix the stale docstring claim that a healthy container proves the port listens.

4. lab.py _step_wait_for_services: skips any Wazuh service the backend already proved in this run (backend.authenticated_readiness[service] is True), so it no longer authenticates twice.

   • Services the backend did not prove keep ADR-030's existing telemetry classification ("do not make every warning fatal"; Reused Wazuh indexer volumes silently keep old admin credentials after .env is changed #623 DEGRADED_USABLE contract).
   • On timeout, the manager diagnostic names the classified phase and reason from one final probe.
   • Deviation from preflight decision 6: fail-closed for this path would reverse ADR-030/Reused Wazuh indexer volumes silently keep old admin credentials after .env is changed #623 policy and is not required by the issue. Persistent failure on the real start path already fails closed in item 3.

5. Docs: add a troubleshooting entry for the new terminal readiness message, and fix the stale curl_status docstring (-u user:pass).

Acceptance → path → evidence

Criterion	tdd_path	Verification
Clean boot emits no avoidable TLS failures during warm-up	B	tests/test_services.py regression (exit 35 → not ready, no WARNING); stateful-loop test: warm-up exit 35 then ready → success, no warnings; live clean aptl lab start from a fresh aptl lab init dir with a transcript free of curl_safe: curl exit
Persistent TLS/authentication failure still fails startup with an actionable diagnostic	B	Stateful tests: exit 35 until the deadline → LabResult failure naming transport/tls_handshake/curl exit 35; 401 until the deadline → authentication/credentials_rejected; malformed status → manager_status; live probe against a published port with no listener
Preserve bounded retries and fail-closed terminal behavior	B	Sleep-count and deadline assertions with the injected clock; no retries inside the probe
Secrets never in argv, logs, results, or errors; temp files removed	B	tests/test_curl_safe.py for curl_request (argv, @file, unlink on success, timeout, and OSError); outcome repr has no token
Lab step does not re-authenticate backend-proved services	B	tests/test_lab.py: proved map → no probe; unproved → existing telemetry path with classified message
Linux and macOS test runners	—	Deterministic unit tests run in the checks.yml OS matrix

Cross-cutting

• Security: reuses the curl_safe 0600 header files; credential-free transport probe; no new argv or log exposure; ADR-034's Wazuh-only -k stays scoped.
• Maintainability: extends the existing helpers; no new retry loop or config key.
• Extensibility: the path, method, and predicate phases sit behind one probe seam.
• Whole-repo: generated realization Compose (the normal path) and the lab step are both covered. Generated .aptl/realization files are not edited.

No requirement is in scope; this is a requirement-free bug fix. No CHANGELOG or version edits. PR title: fix(readiness): ….

[Brad-Edwards]

Live validation (branch 1002-wazuh-api-tls-warmup, uncommitted tree)

Clean boot: wheel built from the branch, fresh venv, fresh aptl lab init directory, aptl lab start.

• Exit 0 with Lab is ready. in 12m53s. All 32 project containers were running.
• Zero curl_safe / curl exit lines in the transcript (previously 3–4 per boot on both install paths).
• The warm-up window was present in this run. aptl-wazuh-manager started at 05:08:57 with no healthcheck (health=none). The API generated its server cert at 05:09:18 and logged Listening on … :55000 at 05:09:22, leaving a 25s non-listening window that the backend readiness loop now waits out quietly.
• The later "Waiting for Wazuh services" step no longer re-authenticates the backend-proved services.
• Teardown with aptl lab stop -v -y exited 0, leaving no aptl-* containers or volumes.

Probe classification against real Wazuh 4.12.0:

• Standalone manager: transport phase failed: tls_handshake (curl exit 35) for 16.6s, then ready, with zero WARNING records.
• Published port with no listener: tls_handshake (curl exit 35). Closed port: connection_refused (curl exit 7).
• Running lab manager with a wrong password: authentication phase failed: credentials_rejected (HTTP 401). With .env credentials: ready.

[Brad-Edwards]

gc_codex_review — cycle 1 of 1 (pre-push) on issue #1002 (branch 1002-wazuh-api-tls-warmup)
Diff mode: inline — the complete diff was supplied in one prompt

Core review

Verdict: don't-ship

The change chooses the right main seams: curl_safe owns classified transport outcomes, services owns the phased manager probe, the deployment backend owns graph-realization readiness, and lab orchestration consumes recorded evidence instead of duplicating successful checks. However, the lifecycle shape is internally inconsistent: the unproved-service fallback still converts terminal failures into warning-only success, while the backend poll repeatedly re-authenticates services it has already proved. Those are authority and state-machine defects in the core readiness contract, not local polish.

Blocking findings (2):

1. [class] Unproved Wazuh services still fail open — src/aptl/core/lab.py:1944
   When the backend has not recorded True, both fallback helpers can exhaust their readiness windows and only emit warning diagnostics; _step_wait_for_services then unconditionally returns None, which lets orchestrate_lab_start continue. The manager path additionally accepts a fresh successful probe after wait_for_service has already reached its deadline, erasing the terminal result. This contradicts the new preflight's single-budget, last-state-at-deadline, and fail-closed requirements. The added tests pin the wrong contract by asserting None after a persistent manager failure and no diagnostic for success after the deadline. Make every supported unproved-service path return a fatal LabResult on persistent failure, or remove the fallback if it is unsupported.
2. [one-off] Latch each service once readiness succeeds — src/aptl/core/deployment/_compose_stateful_readiness.py:204
   Every polling round rebuilds observations for every service. If the indexer succeeds while the manager is warming up, the indexer is authenticated again on every subsequent round; a later transient failure can overwrite its proven success and fail the realization at the manager's deadline. This also directly violates the added preflight's instruction not to recheck a service after it is proven ready. Preserve successful observations and poll only the unresolved services.

Security review

Verdict: ship

This is shaped correctly: it extends the existing curl-safe boundary with a classified, secret-conscious transport outcome, layers phased Wazuh readiness above it, preserves bounded polling and fail-closed backend startup, and keeps public diagnostics limited to phase and numeric status metadata. The design seam supports future readiness phases without duplicating credential or subprocess handling. I found no concrete, exploitable security regression in the supplied diff.

No blocking findings.

[Brad-Edwards]

gc_codex_review pre-push cycle 1 of 1 complete for issue #1002 on branch '1002-wazuh-api-tls-warmup'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next gc_codex_review (uncommitted) invocation to count cycles.

[Brad-Edwards]

Review decision record — codex cycle 1 (issue #1002)

Reviewer: codex
Cycle: 1

Architectural read:

Core reviewer: The change chooses the right main seams: curl_safe owns classified transport outcomes, services owns the phased manager probe, the deployment backend owns graph-realization readiness, and lab orchestration consumes recorded evidence instead of duplicating successful checks. However, the lifecycle shape is internally inconsistent: the unproved-service fallback still converts terminal failures into warning-only success, while the backend poll repeatedly re-authenticates services it has already proved. Those are authority and state-machine defects in the core readiness contract, not local polish.

Security reviewer: This is shaped correctly: it extends the existing curl-safe boundary with a classified, secret-conscious transport outcome, layers phased Wazuh readiness above it, preserves bounded polling and fail-closed backend startup, and keeps public diagnostics limited to phase and numeric status metadata. The design seam supports future readiness phases without duplicating credential or subprocess handling. I found no concrete, exploitable security regression in the supplied diff.

Blocking findings: 2

Finding 1 — class (2 instances)

• ID: F1
• Title: [core] Unproved Wazuh services still fail open
• Location: src/aptl/core/lab.py:1944
• Decision: fix
• Rationale: When the backend has not recorded True, both fallback helpers can exhaust their readiness windows and only emit warning diagnostics; _step_wait_for_services then unconditionally returns None, which lets orchestrate_lab_start contin…
• Instances:
  • src/aptl/core/lab.py:1944
  • src/aptl/core/lab.py:1946

Finding 2 — one-off

• ID: F2
• Title: [core] Latch each service once readiness succeeds
• Location: src/aptl/core/deployment/_compose_stateful_readiness.py:204
• Decision: fix
• Rationale: Every polling round rebuilds observations for every service. If the indexer succeeds while the manager is warming up, the indexer is authenticated again on every subsequent round; a later transient failure can overwrite its proven success …

[Brad-Edwards]

Execution obligation WAZUH-FALLBACK-FAIL-OPEN — Opened

Category: defect
Observed state: When the RAES backend did not prove a Wazuh service's authenticated readiness, _step_wait_for_services in lab.py polls it itself. On a persistent failure it only emits a TELEMETRY warning and startup continues as DEGRADED_USABLE. On this branch the manager path also re-probes once after the deadline, and a late success there erases the deadline result.
Impact: For catalog scenarios and SSH-remote labs, a Wazuh indexer or manager that never becomes ready yields a degraded-but-successful start instead of a failure.
Current obligation: Make the fallback's terminal behavior consistent with the chosen readiness policy. Base diagnostics on the last in-budget observation, never on a post-deadline probe. Apply one classification to indexer and manager alike.

Evidence

• Codex pre-push review cycle 1, class finding 'Unproved Wazuh services still fail open' (lab.py:1944): Investigate transient Wazuh API TLS handshake retries during clean boot #1002 (comment)
• Fallback is reachable: catalog scenarios techvault-defensive-min / techvault-enterprise-web / techvault-attacker-target declare Wazuh nodes with no generated_artifacts, so the backend proves nothing. SSHComposeBackend rejects generated artifacts, so every SSH-remote Wazuh start also uses the fallback.
• ADR-030 names service readiness timeouts as deliberately non-fatal; its Reused Wazuh indexer volumes silently keep old admin credentials after .env is changed #623 amendment and tests/test_lab.py pin indexer HTTP 401 to DEGRADED_USABLE.

[Brad-Edwards]

Execution obligation WAZUH-FALLBACK-FAIL-OPEN — Escalated

Category: defect
Observed state: The lab.py fallback's Wazuh readiness wait is warning-only (ADR-030 telemetry). It is reachable for catalog TechVault scenarios without generated artifacts and for SSH-remote labs.
Impact: A persistent Wazuh TLS/authentication failure fails startup only for graph-owned services; the fallback path reports a degraded success.
Current obligation: Apply one readiness policy to the fallback, with diagnostics from the last in-budget observation.

Evidence

• Codex pre-push review cycle 1 class finding: Investigate transient Wazuh API TLS handshake retries during clean boot #1002 (comment)
• ADR-030 'service readiness timeouts' deliberately non-fatal; Reused Wazuh indexer volumes silently keep old admin credentials after .env is changed #623 amendment pins indexer 401 to DEGRADED_USABLE

Pause class: significant_architecture_decision
Decision request: (A) The fallback fails closed: a persistent Wazuh indexer/manager readiness failure returns a fatal, classified LabResult, via an ADR-030 amendment that reverses #623's DEGRADED_USABLE for indexer 401. (B) Keep ADR-030 telemetry for the fallback, with diagnostics from the last in-budget observation. The codex class finding is recorded wontfix on your authorization, with an ADR-030 note on why graph-owned services fail closed but fallback services degrade.

This obligation remains open while the decision is pending.

[Brad-Edwards]

Plan revision after pre-push review: Wazuh readiness is fail-closed on every path

Decision (user, 2026-09-14): fail closed. "wazuh has to work or the scenario isn't meeting its goals."

This supersedes plan item 4's "services the backend did not prove keep ADR-030's telemetry classification".

• Lab wait_for_services step: covers Wazuh services the backend did not prove (catalog TechVault scenarios with no Wazuh generated artifacts, and SSH-remote labs). A persistent indexer or manager readiness failure now returns a fatal LabResult naming the service, phase, category, and curl/HTTP code. The Reused Wazuh indexer volumes silently keep old admin credentials after .env is changed #623 aptl lab stop -v recovery guidance is kept for indexer 401/403. First failure wins.
• No post-deadline probe: the failure reason is the last observation inside the budget, on both the lab and backend paths.
• Backend readiness loop: latches each service once it is ready. A service already proven is not probed again while another warms up, so a later transient answer cannot overwrite its proof.
• ADR-030 amendment: "Wazuh Readiness Is Startup-Fatal". Other late startup checks (SSH, MCP build, seeding, snapshot) keep their classification.
• Removed: the now-unused check_indexer_ready and check_manager_api_ready wrappers. describe_indexer_status in services.py is the one indexer classification both paths share.

[Brad-Edwards]

Execution obligation WAZUH-FALLBACK-FAIL-OPEN — Resolved

Category: defect
Observed state: The lab.py fallback's Wazuh readiness wait was warning-only. It was reachable for catalog TechVault scenarios with no Wazuh generated artifacts and for SSH-remote labs.
Impact: Before the fix, a Wazuh service that never became ready on the fallback path produced a degraded-but-successful start.
Current obligation: Apply one fail-closed readiness policy on every Wazuh path, with diagnostics taken from the last in-budget observation.

Evidence

• Codex pre-push review cycle 1 class finding: Investigate transient Wazuh API TLS handshake retries during clean boot #1002 (comment)
• User decision 2026-09-14: fail closed, 'wazuh has to work or the scenario isn't meeting its goals'
• Plan revision: Investigate transient Wazuh API TLS handshake retries during clean boot #1002 (comment)

Disposition: fix
Corrective action: The lab _step_wait_for_services now returns a fatal LabResult for a persistent indexer or manager readiness failure. The reason comes from the last in-budget observation (service, phase, category, curl/HTTP code), and #623 recovery guidance is kept for indexer 401/403; there is no post-deadline probe. The backend loop latches services once ready. ADR-030 is amended ("Wazuh Readiness Is Startup-Fatal"), dead boolean wrappers are removed, and the troubleshooting doc is updated.

Verification

• tests/test_lab.py wait_for_services tests: no-response/401/403/manager TLS/credential-rejection failures are fatal, the reason is the last in-budget attempt, the probe count equals the in-budget attempts, and the fatal result maps to StartupOutcome.FAILED
• tests/test_deployment_stateful_realization.py::test_a_proven_service_is_not_reprobed_while_another_warms_up
• gc_implement_mechanical verify on the post-fix tree: completion command (full unit suite + TechVault static gate) passed; make policy passed

[Brad-Edwards]

gc_codex_review — cycle 2 of 1 (pre-push) on issue #1002 (branch 1002-wazuh-api-tls-warmup)
Diff mode: inline — the complete diff was supplied in one prompt

Core review

Verdict: don't-ship

The change chooses the right local layering—classified curl transport, service-level semantic probes, bounded readiness owners, and fatal LabResult propagation—and latching backend evidence is the correct seam for avoiding duplicate authentication. The shape is not yet production-ready, however: probe execution escapes the DeploymentBackend boundary and therefore targets the wrong machine for SSH-remote labs, the phased transport outcome is applied only to the manager while the indexer still discards the evidence this issue requires, and both polling paths can probe and accept success after their declared deadline. These are cross-cutting contract failures rather than local polish, and they foreclose the supported remote-backend variation.

Blocking findings (3):

1. [class] SSH-remote readiness probes the controller instead of the deployment host — src/aptl/core/lab.py:1932
   The newly fatal fallback path explicitly includes SSH-remote labs, but both lab probes construct https://localhost:<published-port> and invoke the controller-local curl boundary. SSHComposeBackend redirects Docker commands with DOCKER_HOST; it does not tunnel these HTTP requests, and a loopback-published Wazuh port exists on the remote host, not the controller. Consequently a supported SSH-remote Wazuh start normally times out against the wrong machine (or observes an unrelated local listener). Route readiness through a typed DeploymentBackend operation or an equivalent backend-owned remote/tunnel mechanism, and cover the SSH backend contract for both services.
2. [class] Indexer readiness discards the transport classification required by the new contract — src/aptl/core/services.py:130
   Only the manager uses CurlOutcome. The indexer still calls curl_status, which returns int | None and ignores curl's return code; describe_indexer_status then turns every refusal, timeout, reset, and TLS failure into no_http_response. A persistent indexer exit 35 therefore cannot produce the promised tls_handshake (curl exit 35) terminal reason, and this path also sends credentials before establishing the credential-free transport phase. Use the classified request outcome for the indexer owners as well, preserving HTTP status and numeric curl exit through the last in-budget observation; update the tests that currently codify the lossy result.
3. [class] Readiness can probe and succeed after the bounded deadline — src/aptl/core/deployment/_compose_stateful_readiness.py:220
   Both Wazuh polling implementations enforce the deadline only after invoking the next probe. If the remaining budget is shorter than the fixed sleep interval, the sleep crosses the deadline and another probe still runs; an after-deadline success is then accepted. Manager attempts can perform three sequential ten-second curl requests, so the overrun is materially larger than the interval. This also means the reported terminal observation need not be the last one made inside the budget. Check the deadline before every retry and clamp the sleep/request budget to the remaining time. Add deterministic cases where the remaining time is shorter than the interval and where a probe would succeed only after expiry.

Security review

Verdict: ship

This is shaped correctly: the change extends the canonical curl boundary with a secret-safe classified outcome, builds phased Wazuh readiness above it, and keeps retry policy at the lifecycle owners. It preserves list-form subprocess execution, permission-restricted credential files, bounded secret-free diagnostics, and fail-closed startup semantics while avoiding duplicate authentication for backend-proven services. The seam supports additional authenticated phases or transport classifications without exposing response bodies or credentials through lifecycle envelopes, and it does not foreclose the obvious next variation of another phased service probe.

No blocking findings.

[Brad-Edwards]

gc_codex_review pre-push cycle 2 (USER-AUTHORIZED OVERRIDE past cap 1) complete for issue #1002 on branch '1002-wazuh-api-tls-warmup'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next gc_codex_review (uncommitted) invocation to count cycles.
Override reason: User authorized one over-cap Codex cycle on 2026-09-14. After cycle 1, the fail-closed decision reworked lab-start Wazuh readiness orchestration and amended ADR-030.