Skip to content

podSpec changes on a live InnoDBCluster are silently ignored (26.7.0-2.3.0) #56

Description

@murad-zeynalli

Summary

Editing spec.podSpec on an existing, running InnoDBCluster has no effect. The CRD accepts the change, the operator logs the complete old→new diff and announces that it is submitting patches, and then patches nothing. There is no error, no warning, and no Kubernetes event — the only signal is a single log line whose flags read like success unless you know what they mean.

The result is that a field which appears to configure the cluster silently does not, and a user has no way to tell without reading operator logs and diffing the generated StatefulSet.

Version

chart       mysql-operator 2.3.0
image       container-registry.oracle.com/mysql/community-operator:26.7.0-2.3.0
Kubernetes  v1.33 (DigitalOcean DOKS)
cluster     3 instances + 2 routers, ONLINE and healthy throughout

Reproduction

  1. Have a running, healthy InnoDBCluster.
  2. Add either of these to spec.podSpec and apply:
spec:
  podSpec:
    securityContext:
      seccompProfile:
        type: RuntimeDefault
spec:
  podSpec:
    containers:
      - name: sidecar
        resources:
          requests: { cpu: 20m, memory: 160Mi }
  1. Watch the operator log and the generated StatefulSet.

Observed

The operator sees the change and logs the full old/new spec, then:

[kopf.objects] [INFO] Fields handled. Time to submit the patches to K8s API!
[kopf.objects] [INFO] InnoDBClusterObjectModifier::submit_patches sts_changed=False
                      sts_spec_changed=False len(router_deploy_patch)=0 len(commands)=0

sts_changed=False. The StatefulSet is not modified. I polled for 8 and 9 minutes respectively across two separate attempts: no rollout, no pod restart, cluster ONLINE 3/3 the whole time.

The live StatefulSet keeps its original pod securityContext:

{"fsGroup":27,"runAsGroup":27,"runAsNonRoot":true,"runAsUser":27}

with no seccompProfile, and resources on the mysql container only — and those were applied at cluster creation, not by a later podSpec edit. So "resources are present in the StatefulSet" is not evidence that podSpec edits work; that misled me initially.

Expected

One of:

  1. Apply the change (rolling the members, as other spec changes do); or
  2. Reject it at admission with a clear message; or
  3. At minimum, emit a warning event saying the field was recognised but will not be applied.

Silently accepting and discarding is the worst of the three, because it looks like it worked.

Why it matters

podSpec is the only documented way to influence the generated pods, and several security baselines cannot be met without it:

  • Pod Security Admission restricted cannot be enforced on the namespace — the operator's containers have no seccompProfile and there is no supported way to add one.
  • Config-audit scanners (Trivy AVD-KSV-0104 seccomp, AVD-KSV-0011/0015/0016/0018 resources) report ~57 findings against the operator's own pods on our cluster, none of them closeable from the manifest.
  • Containers shipped without resource requests — sidecar, metrics, the three init containers, and router — are BestEffort for memory and therefore first in the kubelet's eviction order. The sidecar is the operator's own agent; losing it under node pressure during a rejoin is a poor failure mode.

Note on related closed issues

#18 (router podSpec) and #38 (resources for backup pods) were each closed as individual features, which suggests podSpec is wired per-field rather than passed through generically. If that is the intended design, then this request narrows to: make the unsupported case observable — reject or warn rather than silently drop. That alone would have saved a day of investigation here.

Workaround

A namespace-level LimitRange with defaultRequest supplies requests to the operator's containers at admission time. It cannot supply seccompProfile, so PSA restricted remains unreachable.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions