[FEA] Preserve/upgrade-safe manage partitions and NodeSets created after install via the Slinky management platform

[tsungjung411]

Is your feature request related to a problem? Please describe.

In our deployment, Partition and NodeSet instances are not only defined at install time via values.yaml. After the initial install, additional partitions and NodeSet instances get created later through (our simple) Slinky management platform — i.e., created/edited directly against the live CRs, not by re-running helm install/helm upgrade with an updated values.yaml. As a result, these later-added instances are never reflected back into the values.yaml originally used to install the chart.

When we later perform a minor-version upgrade (e.g. v1.1.Z → v1.2.Z) following the documented flow (helm upgrade slurm-operator-crds → helm upgrade slurm-operator → optionally helm upgrade slurm), we're concerned these out-of-band changes will be lost or left inconsistent:

• From reading the chart templates, Controller.spec.extraConf (which carries the rendered partitions block) appears to be fully re-rendered from values.yaml on every helm upgrade slurm and applied as a single string field rather than merged. Any partition added outside of values.yaml would be silently overwritten the next time an upgrade changes anything in that rendered string, with no warning or conflict detection.
• NodeSet instances created after install (not part of the original values.yaml nodesets map) were never part of that Helm release, so helm upgrade never touches them — meaning their container image (e.g. 25.11-ubuntu24.04 → 26.05-ubuntu26.04) would not be upgraded together with the rest of the fleet, and would require manual intervention per NodeSet.

We wish we could use Slurm Operator to manage partitions and NodeSets dynamically (outside the installing values.yaml) in a way that upgrades don't silently clobber or leave inconsistent.

---

Describe the solution you'd like

A documented, upgrade-safe mechanism for partitions and NodeSets that are created/managed outside of the installing values.yaml, specifically:

• A supported way to add partition configuration that is guaranteed not to be overwritten by helm upgrade slurm — e.g. an explicit statement that configFileRefs (pointing to an externally-managed ConfigMap) is the correct pattern for this, as opposed to extraConf, which is fully owned/re-rendered by the chart.
• Guidance or tooling for keeping the container image of out-of-band NodeSet instances in sync with an operator/chart version upgrade, since they fall outside Helm's tracked release resources.
• Ideally, an explicit statement in the upgrade documentation about what is and isn't safe to manage outside of values.yaml, and what guarantees (if any) apply across upgrades.

---

Describe alternatives you've considered

• Using configFileRefs pointing to a separately-managed ConfigMap instead of writing directly to Controller.spec.extraConf, since configFileRefs is only rendered by the chart when .Values.configFiles is set, and so appears to be left untouched by helm upgrade when unused — but we are not certainsed pattern for this use case.
• Manually diffing the live CR state against the values.yaml before every upgrade and back-porting any out-of-band changes into values.yaml — this is operationally fragile and error-prone at scale.

---

Additional context

This came up while reviewing the README's Upgrades section, which documents the v1.Y upgrade flow (helm upgrade slurm-operator-crds → helm upgrade slurm-operator → helm upgrade slurm) but doesn't address CRs that are created or modified outside of the chart's own values.yaml between installs. Hat our management-platform setup if useful.

[vivian-hafener]

Good afternoon @tsungjung411,

A supported way to add partition configuration that is guaranteed not to be overwritten by helm upgrade slurm — e.g. an explicit statement that configFileRefs (pointing to an externally-managed ConfigMap) is the correct pattern for this, as opposed to extraConf, which is fully owned/re-rendered by the chart.

configFileRefs does not provide a path forward here. That field is intended for Slurm configuration files other than slurm.conf. Due to the current expectations within Slurm about how slurm.conf is formatted, we are limited in how we can interact with that file. We do not have plans to to merge into this file in it's current format. Modifying slurm.conf to be a different format (that would make merging possible) would be considered a Slurm Enhancement, and would have to go through the Slurm folks over at https://support.schedmd.com. At this time, controller.spec.extraConf and partitions within the Helm chart are the two places where partition configuration can be provided. The latter would be your best option to include partition configuration for NodeSets created by tooling other than the Helm chart.

Guidance or tooling for keeping the container image of out-of-band NodeSet instances in sync with an operator/chart version upgrade, since they fall outside Helm's tracked release resources.

The Helm chart can only manage NodeSets that it creates. As such, it is our expectation that NodeSets created outside of the Helm chart are managed by external tooling. We do not intend to introduce tooling to Slurm-operator in order to manage or synchronize NodeSet configuration in this way as that would be out of scope for Slurm-operator. Such a change would prescribe "business logic" and may interfere with sites' existing tooling.

Ideally, an explicit statement in the upgrade documentation about what is and isn't safe to manage outside of values.yaml, and what guarantees (if any) apply across upgrades.

The documentation on upgrades only applies to upgrades conducted using the Helm chart, for NodeSets (and other CRDs) deployed with the Helm chart. As Helm is not responsible for managing NodeSets or partitions that it did not create, those upgrade paths may not function for such NodeSets.

In general, if Helm is being used to manage a Slurm-operator cluster deployment, Helm should be considered the source of truth for how that cluster is configured. Changes that are made outside of the Helm chart may be overwritten by upgrades conducted by Helm, as there is no reasonable way (for Helm) to differentiate between intentional changes and configuration drift. Slurm-operator may also be used without the Helm chart if more flexibility for configuration is desired.

Please reach out if you have any further questions.

Best,
Vivian Hafener

[tsungjung411]

Hi Vivian, thank you for the detailed response.

I'd like to clarify my partition configuration request, as I believe there may be a misunderstanding about the proposed mechanism.

What I'm NOT asking for:

• I'm not asking Slurm itself to support reading slurm.conf from multiple sources
• I'm not asking for any structural "merge" of the slurm.conf format

What I AM proposing:

A new field on the Controller CRD spec — for example spec.extraConfRefs — that accepts a list of ConfigMap references. The operator's reconcile loop would read the contents of those ConfigMaps and concatenate (simple string append) them with spec.extraConf when writing slurm.conf.

final_slurm_conf = base_config + spec.extraConf + concat(spec.extraConfRefs[*])

The resulting slurm.conf is still a single flat text file in the exact format Slurm expects — no parsing, no merging, no format changes. The operator simply assembles it from more than one source before writing.

Why this solves the upgrade-safety problem:

When an external platform creates a NodeSet (outside the Helm release), its corresponding PartitionName=... lines can live in a separate ConfigMap that is not owned by the Helm release. helm upgrade would re-render and overwrite the Controller CR's spec.extraConf from values.yaml, but the externally-managed ConfigMap referenced by spec.extraConfRefs would remain untouched. The operator then assembles the final slurm.conf from both.

There is also a secondary issue worth noting:

The current Helm chart template in templates/controller/_helpers.tpl validates that every NodeSet listed in partitions[x].nodesets must exist as an enabled NodeSet within the same Helm release — and raises a fail if not. This means it's currently impossible to define a partition that spans a Helm-managed NodeSet and an externally-created NodeSet, even through extraConf. Relaxing this validation (or supporting an allowExternalNodeSets: true flag) would be a useful companion change.

Implementation scope:

1. Add extraConfRefs (or similar) to the Controller CRD spec — a []corev1.ConfigMapKeySelector or similar type
2. In the operator's reconcile loop, fetch and append the referenced ConfigMap contents when building slurm.conf
3. Optionally, expose controller.extraConfRefs in the Helm chart values.yaml for declarative usage

This is a contained change to the operator's reconcile logic and the CRD schema. Slurm itself sees no difference in the resulting slurm.conf.

Would the Slinky team be open to considering this as a feature enhancement?

Best,
TJ Tsai

[vivian-hafener]

Good afternoon @tsungjung411,

Both of those suggestions sound reasonable to me, though further discussion with the rest of the team would be needed to determine if they align with our long-term goals.

For the Slinky team to implement these enhancements in-house, we would need you to submit an enhancement request at support.schedmd.com.

Alternatively, if you would like to submit PRs to add these features (one PR for each feature), feel free to do so. We try to read through most of the PRs as a team, and pick ones that align with our long-term goals to accept and merge.

Best,
Vivian Hafener