Phase 2: k3d functional test for reconcile-on-startup unblocks stuck delete

[nithyatsu]

Summary

Add a k3d functional test that proves the full rad startup → reconcile → rad app delete loop unblocks a stuck non-terminal resource, exactly as described in specs/006-state-restoration/plan.md §Phase 2.

Context

Phase 1 (PR #12871) shipped:

• Unit tests on the dynamic-rp reality-check handler (gone → Failed, settled → Succeeded, cloud output → skipped, terminal → empty).
• Unit tests on the corerp orchestrator (fan-out, terminal skip, dispatch failure recorded).
• An httptest-backed integration test that exercises the real HTTP dispatch path against a fake UCP + downstream RP.
• Unit tests on the new ReconcileHydratedState stage in rad startup.

What Phase 2 adds is confidence that the full loop works against a real k3d cluster + real Kubernetes + real OCI state archive — not new correctness coverage, but a regression fence that catches drift as the surrounding code evolves.

Acceptance

An E2E test in test/functional-portable/statestore/noncloud/ that:

1. Installs Radius with the OCI state backend enabled (reuses installRadius + startLocalRegistry already in that package).
2. Deploys a minimal Radius.Compute/containers application.
3. rad shutdown — produces an OCI archive.
4. Poisons the archive — opens it via pkg/statearchive/factory, rewrites the container's provisioningState to "Updating", re-saves.
5. rad uninstall --purge → rad install — fresh cluster.
6. rad startup — restores the poisoned state, then the new ReconcileHydratedState stage fires, dynamic-rp GETs the container's outputResources (the k8s Deployment doesn't exist), aggregates to Failed, PATCHes provisioningState.
7. Asserts the container's provisioningState is Failed via rad resource show.
8. rad app delete <name> --yes --preview succeeds without a 409 loop.

Unknown to resolve first

The step-4 "poison the archive" primitive doesn't exist yet as a test helper. Options depending on the archive Session interface's ergonomics:

• A: use pkg/statearchive/factory to open the archive, iterate keys, find the dynamic-rp row for the container, mutate JSON, save. ~150 LOC helper.
• B: if the OCI backend is write-once, unpack the archive manually (OCI descriptor → tarball → JSON files), edit, repack. ~300 LOC helper.
• C: (rejected in PR [Spec] Reconcile hydrated state against reality on rad startup #12871 planning) add a test-only POST /debug/force-provisioning-state route on dynamic-rp gated by RADIUS_ENABLE_TEST_ENDPOINTS=true. Smaller test code, adds a test-only surface to the production RP.

Pick a direction after a short spike on the archive Session API.

Related

• PR [Spec] Reconcile hydrated state against reality on rad startup #12871 — the Phase 0 + Phase 1 implementation this follows up on.
• Verify rad app delete is idempotent under concurrent invocations against the same application #12870 — separate follow-up on concurrent rad app delete behavior.