`rad app delete --preview` succeeds while recipe-generated Kubernetes resources remain

[willdavsmith]

Summary

rad app delete <application> --yes --preview reports successful deletion of the application's Radius resources and application record, but the Kubernetes resources generated by those resources' recipes remain on the target cluster.

This was reproduced from a clean application baseline using an ephemeral Radius control plane with state restored by rad startup. Both container and routes outputs remained, so the problem is not specific to Gateway API cleanup.

Reproduction

1. Remove all existing Kubernetes resources labeled radapp.io/application=gateway-managed-demo.

2. Deploy an application containing:

   • Radius.Compute/containers resource web
   • Radius.Compute/routes resource web
   • Radius.Compute/containerImages resource web-image
   • Radius.Security/secrets resource radius-ghcr-registry-creds

3. Verify that fresh Kubernetes resources were created on the target AKS cluster.

4. Persist the Radius control-plane and recipe state with rad shutdown.

5. Start a new ephemeral control plane and restore the state with rad startup.

6. Run:

   rad app delete gateway-managed-demo --yes --preview

7. Inspect the target cluster:

   kubectl get deployments,services,httproutes,tcproutes,tlsroutes,udproutes \
     -A -l radapp.io/application=gateway-managed-demo

Expected behavior

The resource delete operations should remove their recipe-generated Kubernetes outputs before the long-running operations complete and before the application deletion reports success.

Actual behavior

The CLI reported:

Deleting 4 resource(s) associated with application gateway-managed-demo...
  Deleting .../Radius.Compute/containers/web...
  Deleting .../Radius.Compute/routes/web...
  Deleting .../Radius.Compute/containerImages/web-image...
  Deleting .../Radius.Security/secrets/radius-ghcr-registry-creds...
Deleting application gateway-managed-demo...
Application deleted

However, the following freshly created resources remained after the workflow completed:

• Deployment default/web
• Service default/web-web
• HTTPRoute default/routes-ly4cicrlqtmbo
• TCPRoute default/routes-ly4cicrlqtmbo
• TLSRoute default/routes-ly4cicrlqtmbo
• UDPRoute default/routes-ly4cicrlqtmbo

The clean deploy created these objects around 2026-08-31T20:59Z; the delete completed around 2026-08-31T21:05Z, and the objects still existed afterward.

Workflow evidence

• Clean deploy: https://github.com/willdavsmith/radius-gateway-managed-demo/actions/runs/33438499080
• Delete reproduction: https://github.com/willdavsmith/radius-gateway-managed-demo/actions/runs/33439054569

The delete workflow logged successful restoration of both control-plane databases and Terraform recipe state before executing the command.

Impact

This leaves application workloads and routing resources orphaned while Radius no longer has the application/resource records needed to delete them on a subsequent run. It also prevents safe shared Gateway cleanup: the Gateway lifecycle action sees the remaining Route objects and correctly retains the Gateway, GatewayClass, Contour installation, and Gateway API CRDs.

Investigation notes

DeleteResourcesInParallel waits for each management client's delete poller, so the unresolved boundary appears to be between the Radius resource deletion completing and its recipe-generated Kubernetes outputs being removed. Possible areas include restored recipe-state association, Bicep recipe deletion, or Kubernetes output deletion. The workflow evidence does not yet distinguish among them.

[radius-triage]

👍 We've reviewed this issue and have agreed to add it to our backlog. Please subscribe to this issue for notifications, we'll provide updates when we pick it up.

We also welcome community contributions! If you would like to submit a pull request, please visit our contribution guidelines and assign this to yourself by commenting "/assign" on this issue.

For more information on our triage process please visit our triage overview

[nicolejms]

Please repo the issue.

[nicolejms]

This reproduces consistently in the ai-extensions Cloud E2E suite — every run, on a two-cluster topology. Full write-up in radius-project/ai-extensions#849. Analysis below; the short version is that the Bicep recipe delete path never became multi-cluster aware, so the DELETE is sent to the wrong cluster and a 404 is swallowed as success.

Topology

The Radius control plane runs in an ephemeral per-run k3d cluster; workloads are deployed to a separate long-lived AKS cluster via RADIUS_TARGET_KUBECONFIG. Only Radius.Compute/containers and Radius.Data/postgreSqlDatabases — no routes, no Gateway, so this is not Gateway-specific.

Evidence

rad app delete returns exit 0 and empties the resource graph, but the workloads are never touched. Timeline from one run, with a watcher polling the AKS cluster throughout:

Time (UTC)	Event
01:51:16	rad: Deleting …Radius.Compute/containers/sleeper
01:52:52	rad: "Application deleted" — exit 0, 96s later
01:53:03	rad app list returns empty
01:53:07	ephemeral k3d control plane destroyed
01:55:25	Deployment/sleeper still 1/1 READY, age 14m
01:59:25	test's 30s absence assertion times out

Object metadata at 01:55, while the objects still existed:

• deletionTimestamp: absent — no delete was ever issued against them
• finalizers: none — nothing was blocking removal
• ownerReferences: none — so Kubernetes garbage collection can never reclaim them either
• labels intact: radapp.io/application, radapp.io/environment, radapp.io/resource

With no owner reference, the only thing that could ever remove these objects is an explicit call from Radius, and that call never came. They leak permanently. A longer wait does not help.

Root cause

Multi-cluster v1 (#12106, first shipped in v0.60.0) taught the deploy path to target an external cluster via RADIUS_TARGET_KUBECONFIG, but the Bicep recipe-delete path still resolves the control-plane cluster:

1. Radius.Compute/containers declares no ManualResourceProvisioning, so dynamic-rp routes the delete to the recipe delete controller (pkg/dynamicrp/backend/controller/dynamicresource.go).
2. That reaches bicepDriver.Delete (pkg/recipes/driver/bicep/bicep.go), which deletes each output resource through resourceClient.deleteKubernetesResource (pkg/portableresources/processors/resourceclient.go).
3. That client comes from kubernetesclientprovider.FromOptions(config.Kubernetes) (pkg/dynamicrp/options.go) — the control-plane provider, never the injected target. RADIUS_TARGET_KUBECONFIG is only consumed by pkg/recipes/kubernetes/clusteraccess/injected.go (recipe execution) and pkg/server/asyncworker.go (legacy corerp clients).
4. So DELETE Deployment/sleeper lands on the k3d control plane, 404s, and IgnoreNotFound turns the miss into a nil error. The resource record is deleted, the app is deleted, rad exits 0.

Two things compound it:

• The legacy Applications.Core/containers path was made multi-cluster aware in the same PR, via the async worker's target-resolving clients. So legacy delete still cleans up correctly and only the preview path leaks — that asymmetry is probably why this went unnoticed.
• No cluster identity is persisted on Kubernetes output resources. IDs hardcode the plane name local (pkg/ucp/resources/kubernetes/kubernetes.go), so even a corrected delete path cannot recover which cluster an object lives on from the record alone.

Single-cluster deletes are covered and pass (validation.ValidateNoPodsInApplication), which is consistent with this being specific to an external target cluster.

Why CI did not catch it

The multi-cluster test added by #12106 (test/functional-portable/multicluster/noncloud/resources/bicep_container_test.go) verifies creation on the external cluster but performs no post-delete verification there — the K8s object validation is explicitly left empty.

Repro

No rad shutdown/startup needed; state restoration is not involved, since containers use a Bicep recipe and the delete input is status.outputResources, not Terraform state.

1. Install Radius with global.targetCluster.enabled=true pointing at a second cluster.
2. Deploy a Radius.Compute/containers resource using the Kubernetes recipe pack; confirm the Deployment exists on the external cluster.
3. rad app delete <app> --yes → succeeds; the objects remain on the external cluster.

Suggested fixes

1. Resolve the dynamic-rp Bicep driver's ResourceClient against the deployment target cluster, mirroring what pkg/server/asyncworker.go already does for the legacy path.
2. Persist cluster identity on Kubernetes output resources instead of the hardcoded local plane, so deletes are unambiguous.
3. Make the silent successes loud: do not treat IgnoreNotFound as a clean result for a Radius-managed output resource. There is a second, independent silent-success path with the same symptom — pkg/recipes/terraform/execute.go logs "Skipping deletion of recipe resources: Terraform state file backend does not exist" and returns nil — worth hardening at the same time.
4. Add post-delete external-cluster assertions to the multi-cluster functional test.

Unexplained

rad app delete spends ~96s on Radius.Compute/containers/sleeper before reporting success, so something is waiting on something. We could not attribute that to a specific code path; candidates are API discovery in lookupKubernetesAPIVersion, the Bicep delete retry loop, or async-operation poll intervals. Flagging it in case it points somewhere useful.