During a deployment on an elemental created Kubernetes cluster, installation of the gpu-operator Helm chart failed due to incorrect credentials. Consequently, unrelated independent components (local-path-provisioner and ollama) also failed to deploy because their manifests and secrets were never applied. There were no dependencies between these components. Once the credentials were fixed, the entire deployment succeeded.
This behavior seems to stem from how manifests are applied sequentially in the k8s_res_deploy.sh.tpl. Any single failure (such as a bad credential) aborts the entire execution loop. While a fail-fast strategy makes sense for coupled stacks, it acts as an unnecessary blocking mechanism for independent resources, resulting in:
- Cascading Failures: A localized misconfiguration in one component halts the deployment of the entire independent stack.
- Difficult Diagnostics: Healthy, unrelated components are left uncreated, making troubleshooting harder.
- Inconvenient Retries: Users may need to trigger redeployments even for completely decoupled workloads.
A few questions/suggestions for making this phase of the deployment more resilient:
- Is this strict fail-fast behavior intended for all manifests?
- can the script gracefully continue deploying remaining independent manifests and return a consolidated error status at the end?
During a deployment on an
elementalcreated Kubernetes cluster, installation of thegpu-operatorHelm chart failed due to incorrect credentials. Consequently, unrelated independent components (local-path-provisionerandollama) also failed to deploy because their manifests and secrets were never applied. There were no dependencies between these components. Once the credentials were fixed, the entire deployment succeeded.This behavior seems to stem from how manifests are applied sequentially in the k8s_res_deploy.sh.tpl. Any single failure (such as a bad credential) aborts the entire execution loop. While a fail-fast strategy makes sense for coupled stacks, it acts as an unnecessary blocking mechanism for independent resources, resulting in:
A few questions/suggestions for making this phase of the deployment more resilient: