Skip to content

[RFE] Prevent single manifest failure from blocking independent resources #517

Description

@fdegir

During a deployment on an elemental created Kubernetes cluster, installation of the gpu-operator Helm chart failed due to incorrect credentials. Consequently, unrelated independent components (local-path-provisioner and ollama) also failed to deploy because their manifests and secrets were never applied. There were no dependencies between these components. Once the credentials were fixed, the entire deployment succeeded.

This behavior seems to stem from how manifests are applied sequentially in the k8s_res_deploy.sh.tpl. Any single failure (such as a bad credential) aborts the entire execution loop. While a fail-fast strategy makes sense for coupled stacks, it acts as an unnecessary blocking mechanism for independent resources, resulting in:

  • Cascading Failures: A localized misconfiguration in one component halts the deployment of the entire independent stack.
  • Difficult Diagnostics: Healthy, unrelated components are left uncreated, making troubleshooting harder.
  • Inconvenient Retries: Users may need to trigger redeployments even for completely decoupled workloads.

A few questions/suggestions for making this phase of the deployment more resilient:

  • Is this strict fail-fast behavior intended for all manifests?
  • can the script gracefully continue deploying remaining independent manifests and return a consolidated error status at the end?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions