[ray-operator] Add idleTerminationOptions for idle RayCluster termination

[justinyeh1995]

Why are these changes needed?

The Ray Autoscaler can scale worker pods down to zero based on idleTimeoutSeconds, but the RayCluster custom resource and its associated resources still remain in the cluster. As a result, the head pod continues to consume resources and may reserve capacity even when the cluster is otherwise idle. Users who do not want to retain the head pod or its reserved capacity must manually delete the RayCluster.

With ray-project/ray#65763, the Ray Autoscaler automatically terminates an idle RayCluster when it has no attached user driver. This PR aims to implement KubeRay's side of the idle termination feature by introducing two apis, which are idleSuspend and idleTerminationOptions.

When idleTerminationOptions is set, the ray autoscaler will terminate an idle RayCluster. If the idleTerminationOptions.policy is Suspend, the ray autoscaler will patch idleSuspend=true, and the reconcile loop will recognize it and suspend the cluster same as how spec.Suspend works. The only difference is the reason emit in the event.

On the other hand, when idleTerminationOptions.policy is Delete, the reconcile loop will log the event and remove the finalizer to allow the deletion.

Changes:

API and Constants

• Add IdleSuspend to RayClusterSpec.
• Add TimeoutSeconds and Policy to IdleTerminationOptions.
• Add the ray.io/no-driver-idle-termination finalizer and RayClusterIdleSuspended condition reason.

Reconciliation

• Treat IdleSuspend as a suspension trigger and reuse the existing RayCluster suspend flow.
• Set RayClusterIdleSuspended as the condition reason when suspension is triggered by idle termination.
• Handle the idle-termination finalizer during deletion and remove it after recording the corresponding logs and events.

RBAC

• Grant the autoscaler delete permission only for the target RayCluster using resourceNames.

Validation and Compatibility

• Require Autoscaler v2 and enableInTreeAutoscaling when IdleTerminationOptions.TimeoutSeconds is configured.
• Validate the timeout and policy configuration.
• Exclude RayJob, RayService, Ray versions earlier than 2.56.
  (will be bumped to 2.59.0 once [Autoscaler] Support idleTerminationOptions for RayCluster idle termination ray#65763 is merged)

Related issue number

Closes #2998
Related PR #4932
Ray ray-project/ray#65763

Labels

• If this PR has user-facing changes that require documentation updates at release time, I have added the doc-updates-required label.
• If this PR contains breaking changes, I have added the breaking-change label.

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests with steps listed in [ray-operator] Add idleTerminationOptions for idle RayCluster termination #5210 (comment)
  • TODO: the e2e test will be added once the ray's idle termination pr is merged.
  • This PR is not tested :(

[justinyeh1995]

Manual testing

1. Setting up the environment

Create a test cluster

TEST_CLUSTER="idle-termination-cluster"
kind create cluster --name "$TEST_CLUSTER" --image="kindest/node:v1.35.0"

Build & Load the Ray Image into cluster

Since the idleTermination feature is still in review, we build a local image and then manually load it into k8s cluster. We are targeting getting this feature into Ray 2.59.0

cd ray # go to ray root directory
gh pr checkout 65763

# patch the changed files to the base ray:2.58.0 image
cat <<'EOF' > Dockerfile.idle-termination
FROM rayproject/ray:2.58.0
COPY ray/autoscaler/v2/instance_manager/cloud_providers/kuberay/cloud_provider.py \
     /home/ray/anaconda3/lib/python3.10/site-packages/ray/autoscaler/v2/instance_manager/cloud_providers/kuberay/cloud_provider.py
COPY ray/autoscaler/_private/kuberay/node_provider.py \
     /home/ray/anaconda3/lib/python3.10/site-packages/ray/autoscaler/_private/kuberay/node_provider.py
EOF

docker build -t rayproject/ray:idle-termination -f Dockerfile.idle-termination python/

kind load docker-image rayproject/ray:idle-termination --name "$TEST_CLUSTER"

Build & Load the kuberay-operator image

cd kuberay/ray-operator
gh pr checkout 5210
IMG=kuberay/operator:idle-termination make docker-build

kind load docker-image kuberay/operator:idle-termination --name "$TEST_CLUSTER"

Install the deployment with helm

cd .. # kuberay/
helm install kuberay-operator ./helm-chart/kuberay-operator \
  --set image.repository=kuberay/operator \
  --set image.tag=idle-termination \
  --set image.pullPolicy=IfNotPresent

---

2. Testing idleTerminationOptions

Testing Suspend Policy

1. Verify the RayCluster change from running to suspended after spec.idleTerminationOptions.timeoutSeconds and the spec.idleSuspend=true. The reason should also be RayClusterIdleSuspended.
2. Manually patch spec.idleSuspend=false to check if the RayCluster resumes
3. Run a driver ray job in the head container, and observe it only start terminating after the job is done.

# Please make sure the current directory is the root directory before proceeding
kubectl apply -f ray-operator/config/samples/ray-cluster.autoscaler-v2-terminate-idle-cluster.yaml

# the ray-cluster.autoscaler-v2-terminate-idle-cluster.yaml named the RayCluster as ray-cluster-autoscaler-v2-terminate-idle-cluster
RAY_CLUSTER=ray-cluster-autoscaler-v2-terminate-idle-cluster

kubectl wait --for=jsonpath='{.status.state}'=ready raycluster/"$RAY_CLUSTER" --timeout=300s

kubectl get raycluster "$RAY_CLUSTER" -o jsonpath='{.spec.idleTerminationOptions.policy}'

After spec.idleTerminationOptions.timeoutSeconds, check if the RayCluster has been idleSuspended with the correct reason. The pods should also be deleted.

kubectl get raycluster "$RAY_CLUSTER" -o jsonpath='{.status.state}{"\n"}'
# suspended

kubectl get raycluster "$RAY_CLUSTER" -o jsonpath='{.status.conditions[?(@.type=="RayClusterSuspended")].status}{" "}{.status.conditions[?(@.type=="RayClusterSuspended")].reason}{"\n"}'
# True RayClusterIdleTerminated

kubectl get pods -l ray.io/cluster="$RAY_CLUSTER"
# No resources found in default namespace.

---

Now, verify Suspend Policy with a driver running for 90s. After around 90s and another 30s, it will get idle suspended.

This step requires 3 different panels for observability

• Panel A

#manually merge-patch  spec.idleSuspend=false to resume the idleSuspended RayCluster
kubectl patch raycluster "$RAY_CLUSTER" --type merge -p '{"spec":{"idleSuspend":false}}'
kubectl wait --for=jsonpath='{.status.state}'=ready raycluster/"$RAY_CLUSTER" --timeout=120s

#exec into ray head container, and run a no-op rayjob
HEAD_POD=$(kubectl get pods -l ray.io/cluster="$RAY_CLUSTER",ray.io/node-type=head -o jsonpath='{.items[0].metadata.name}')

kubectl exec -it "$HEAD_POD" -c ray-head -- ray job submit --address http://127.0.0.1:8265 --submission-id rayjob-as-a-driver --no-wait -- python -c  "
import ray, time
ray.init()
print('sleep started at:', time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()))
time.sleep(90)
print('sleep ended at:', time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()))
"

kubectl exec -it "$HEAD_POD" -c ray-head -- ray job logs rayjob-as-a-driver --follow --address http://10.244.0.8:8265

• Panel B

kubectl get raycluster "$RAY_CLUSTER" -w

• Panel C

kubectl get events --field-selector involvedObject.name="$RAY_CLUSTER" -o jsonpath='{range .items[*]}{.eventTime}{"\t"}{.series.lastObservedTime}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

Testing the Delete Policy

1. Verify the RayCluster CR is deleted after spec.idleTerminationOptions.timeoutSeconds with no driver. The event should log the reason.
2. Verify the RayCluster CR got deleted by querying about the underlying Pods

First, manually update the sample yaml. to spec.idleTerminationOptions.policy = "Delete" then apply it

• Panel A

sed -i 's/policy: Suspend/policy: Delete/' ray-operator/config/samples/ray-cluster.autoscaler-v2-terminate-idle-cluster.yaml
kubectl apply -f ray-operator/config/samples/ray-cluster.autoscaler-v2-terminate-idle-cluster.yaml

• Panel B

kubectl get raycluster "$RAY_CLUSTER" -w -o jsonpath='{.metadata.deletionTimestamp}{"\n"}{.metadata.finalizers}{"\n"}'

• Panel C

kubectl get raycluster "$RAY_CLUSTER" -w

kubectl get events --field-selector involvedObject.name="$RAY_CLUSTER" -o jsonpath='{.items[0].reason}{"\n"}{.items[0].message}'

kubectl get pods -l ray.io/cluster="$RAY_CLUSTER"

3. Clean Up

kind delete cluster --name "$TEST_CLUSTER"

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Disabled feature leaves finalizer stuck

Medium Severity

The operator removes ray.io/no-driver-idle-termination only when IsIdleTerminationOptionsEnabled is true. If that finalizer is present but autoscaling or idleTerminationOptions has been turned off, deletion skips cleanup and the RayCluster stays Terminating forever.

 

^{Reviewed by Cursor Bugbot for commit 46bb176. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{Reviewed by Cursor Bugbot for commit 0b0eae5. Configure here.}

[cursor]

Existing Roles omit delete permission

Medium Severity

BuildRole now grants delete (and scopes the rule with resourceNames), but reconcileAutoscalerRole still leaves an existing Role unchanged. Clusters created before this version never receive delete, so setting idleTerminationOptions.policy to Delete after upgrade fails with a 403 and the idle cluster is not removed.

 

^{Reviewed by Cursor Bugbot for commit 0b0eae5. Configure here.}