Skip to content

fix(aws-eks-cluster): drop cilium startup taint from the bootstrap NodePool - #926

Merged
alexlokshin-czi merged 2 commits into
mainfrom
fix/cilium-taint-bootstrap-deadlock
Sep 10, 2026
Merged

alexlokshin-czi merged 2 commits into
mainfrom
fix/cilium-taint-bootstrap-deadlock

Conversation

@alexlokshin-czi

@alexlokshin-czi alexlokshin-czi commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Declaring node.cilium.io/agent-not-ready on the bootstrap NodePool makes a fresh cluster unable to finish its first apply. Only cilium removes the taint; cilium arrives via ArgoCD; ArgoCD needs the cluster registered; registration needs a completed apply. dev-marcom and prod-marcom hit this and failed four times (run) — cert_manager timed out on its post-install hook after about 21 minutes, taking 10 downstream terragrunt units with it.

Two changes break the cycle:

  • The Terraform NodePool no longer declares the taint. Since the v9 default-NodePool handover, the karpenter-default-nodepool ArgoCD app owns it and applies it from common.yaml, so steady state is unchanged and the double-provisioning protection from feat(aws-eks-cluster): opt-in cilium startup taint on the default NodePool #895 is retained. The NodePool half was create-time only (ignore_changes on yaml_body), so existing clusters see no diff.
  • cert-manager now tolerates the taint. fix!: tolerate the cilium startup taint in Deployment-backed EKS addons #914 covered coredns, aws-ebs-csi-driver, aws-mountpoint-s3-csi-driver and external-secrets but missed cert-manager, which is enabled by default and blocks on a post-install hook. All four subcharts need it: controller, webhook, cainjector and startupapicheck were each Pending on dev-marcom.

karpenter_declare_cilium_startup_taint now means only "this cluster will run cilium, so tolerate its startup taint." Variable description and README updated to match.

Review focus

  • Non-ArgoCD callers. A cluster not registered with the hub now gets no startup taint at all, since nothing else supplies it. Every current caller is hub-registered, so this is about future ones — say the word if you'd rather gate this on a flag than drop it outright.
  • cert-manager toleration paths. Top-level tolerations plus webhook/cainjector/startupapicheck, against chart v1.20.2. Worth a second pair of eyes that these are the right keys and that startupapicheck honours them on the hook Job.
  • Still-uncovered releases. The variable docs continue to warn that other wait = true releases (kubecost, jupyterhub) will deadlock a flag-on fresh cluster. Not addressed here.

Test plan

terraform fmt is clean. terraform validate cannot pass standalone on this module — it errors on the aws.us-east-1 provider alias that callers supply — and its output is byte-identical before and after this change, so it is uninformative rather than passing.

The lint check is red, inherited from main: a Go typecheck failure in scripts/snowflake_generate_grant_all introduced by 52fc695 (Snowflake provider bump) on 2026-09-01. Every main run before that was green. This PR touches no Go.

Real verification is a fresh-cluster apply: re-run the marcom apply and confirm the eks unit completes and the 10 skipped units run.

alexlokshin-czi and others added 2 commits September 10, 2026 18:49
…um startup taint

Declaring node.cilium.io/agent-not-ready on the bootstrap NodePool deadlocks a
fresh cluster: only cilium removes the taint, and cilium arrives via ArgoCD
after the cluster is registered, which requires a completed apply. The
karpenter-default-nodepool ArgoCD app now owns the taint for steady state.

Also tolerate the taint in cert-manager, which is enabled by default and blocks
on a post-install hook, so it was not covered by the existing toleration set.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@alexlokshin-czi
alexlokshin-czi requested a review from a team as a code owner September 10, 2026 18:51
@alexlokshin-czi alexlokshin-czi changed the title fix(aws-eks-cluster): unblock fresh-cluster bootstrap behind the cilium startup taint fix(aws-eks-cluster): drop cilium startup taint from the bootstrap NodePool Sep 10, 2026
@alexlokshin-czi
alexlokshin-czi merged commit 44f3bb5 into main Sep 10, 2026
10 of 11 checks passed
@alexlokshin-czi
alexlokshin-czi deleted the fix/cilium-taint-bootstrap-deadlock branch September 10, 2026 19:02
@czi-github-helper czi-github-helper Bot mentioned this pull request Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants