Make worker (slurmd) SecurityContext configurable / support unprivileged slurmd

[kiarie-kaskazi]

Summary

The worker builder hard-codes a privileged SecurityContext on the slurmd container, unconditionally, with no way to override it from the NodeSet spec or the Helm chart. This makes it impossible to run slurmd unprivileged, which is a hard requirement for MIG-sliced, multi-tenant Kubernetes clusters.

internal/builder/workerbuilder/worker_app.go (v1.2.0), func (b *WorkerBuilder) slurmdContainer(...), lines 249–259:

SecurityContext: &corev1.SecurityContext{
    Privileged: ptr.To(true),
    Capabilities: &corev1.Capabilities{
        Add: []corev1.Capability{
            "BPF",
            "NET_ADMIN",
            "SYS_ADMIN",
            "SYS_NICE",
        },
    },
},

Because this is set on the builder's Base container, a securityContext supplied through the NodeSet's spec.slurmd container template is overridden by the base during the merge — the rendered worker pod is always privileged: true. There is no chart value or operator flag to change it.

Why unprivileged slurmd matters (MIG / multi-tenant)

On a GPU node partitioned into MIG instances, tenants each get a single MIG slice injected by the NVIDIA device plugin (e.g. nvidia.com/mig-1g.18gb: 1). The device plugin scopes the container's /dev and cgroup so the workload sees exactly one MIG device.

A privileged slurmd bypasses that scoping: via NVML it enumerates every MIG instance on the physical GPU, not just the one injected into its pod. Slurm's GRES autodetect then counts more devices than the node is configured to advertise, and the node registers as INVALID / goes down (GRES count mismatch). In other words, privileged slurmd and per-slice MIG scheduling are mutually exclusive.

Unprivileged slurmd works — verified

Running the same slurmd image unprivileged (no privileged, no added capabilities), with:

• an explicit File= gres.conf (e.g. AutoDetect=off + Name=gpu File=/dev/nvidia0) rather than AutoDetect=nvml, and
• cgroup.conf with CgroupPlugin=disabled (plus ProctrackType=proctrack/linuxproc),

a single device-plugin-injected MIG slice:

• reports exactly one device to NVML (nvidia-smi -L → 1 MIG),
• registers healthy (sinfo → idle, no INVALID_REG, no sticky DRAIN), and
• runs sbatch/srun jobs to COMPLETED on that slice.

This is safe because the pod's Kubernetes cgroup and the device plugin already provide CPU/memory/GPU isolation — Slurm's own cgroup enforcement is redundant when one pod maps to one slice. Unprivileged slurmd cannot initialize Slurm's cgroup plugin anyway (/sys/fs/cgroup is read-only in the container, and the systemd/dbus path is unavailable), which is the second reason CgroupPlugin=disabled is needed for this mode.

Today the only way to get an unprivileged worker is to strip the hard-coded SecurityContext out-of-band (e.g. an admission-time mutating policy), which is fragile and defeats the purpose of the operator managing the pod spec.

Request

Please make the worker (slurmd) SecurityContext configurable. Concretely, any of:

1. Honor a NodeSet-supplied securityContext on the slurmd container (merge so the user's value wins over the hard-coded base), and/or
2. Add an explicit unprivileged mode for NodeSets (drop privileged + the added capabilities), optionally defaulting CgroupPlugin=disabled in that mode, and/or
3. At minimum, expose a chart value / operator flag to opt out of the privileged worker.

The privileged default can remain for existing deployments; the ask is a supported path to an unprivileged worker for MIG/multi-tenant Kubernetes.

Happy to open a PR if a preferred shape is indicated (e.g. respecting spec.slurmd securityContext, or a spec.slurmd.unprivileged toggle).

Environment: slurm-operator + slurm chart v1.2.0, Slurm 26.05 images, Kubernetes with the NVIDIA device plugin and MIG-partitioned GPUs.

[vivian-hafener]

Good afternoon @kiarie-kaskazi,

slurmd is expected to be run as root. This is required for slurmd to run jobs as the user that submitted them, which is usually a desired configuration. Running slurmd as a user other than root would, in most cases, weaken the security posture of a cluster as it would mean that all users' jobs would be launched as the same user.

It is also our expectation that in most cases, the Slinky NodeSet pod on a Kubernetes node consumes the whole node - and thus all GPU devices on that node. In such a configuration, Slurm would be responsible for the management of those resources.

Could you explain your use case in more detail?

Best,
Vivian Hafener