MySQL server Pods can fail to start when Kopf writes large last-handled-configuration annotation to Pods

[ymygogogo]

Description

We observed MySQL InnoDBCluster Pods failing to start with MySQL Operator community-operator:8.0.36-2.0.13.

The kubelet event shows containerd rejecting container creation because the generated container labels exceed containerd's 4096-byte key/value size
limit. The failing key is Kopf's default diff-base annotation:

Error: failed to create containerd container: create container failed validation:
containers.Labels: label key and value length (9875 bytes) greater than maximum size (4096 bytes),
key: kopf.zalando.org/last-handled-configuration

Environment

• MySQL Operator image: community-operator:8.0.36-2.0.13
• MySQL server image: community-server:8.0.36
• Custom resource: mysql.oracle.com/v2, kind InnoDBCluster
• Runtime: containerd

What happened

One MySQL server Pod entered BackOff because containerd refused to create the mysql container. The InnoDBCluster custom resource did not set
spec.podAnnotations or spec.podLabels, so this annotation was not user-provided.

Workaround

Removing the annotation from the affected Pod allowed the container to start successfully:

kubectl annotate pod -n base mysql-innodbcluster-0 kopf.zalando.org/last-handled-configuration-

Suspected cause

The operator registers Kopf handlers directly on MySQL server Pods:

@kopf.on.create("", "v1", "pods", labels={"component": "mysqld"})
@kopf.on.event("", "v1", "pods", labels={"component": "mysqld"})
@kopf.on.delete("", "v1", "pods", labels={"component": "mysqld"})

Kopf appears to persist its diff-base state on the handled Pod using:

kopf.zalando.org/last-handled-configuration

For large Pods, this annotation can become large enough that kubelet/containerd receives a label key/value pair above containerd's 4096-byte limit,
preventing container creation.