fix(aws-eks-cluster): drop cilium startup taint from the bootstrap NodePool - #926
Merged
Merged
Conversation
…um startup taint Declaring node.cilium.io/agent-not-ready on the bootstrap NodePool deadlocks a fresh cluster: only cilium removes the taint, and cilium arrives via ArgoCD after the cluster is registered, which requires a completed apply. The karpenter-default-nodepool ArgoCD app now owns the taint for steady state. Also tolerate the taint in cert-manager, which is enabled by default and blocks on a post-install hook, so it was not covered by the existing toleration set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jakeyheath
approved these changes
Sep 10, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Declaring
node.cilium.io/agent-not-readyon the bootstrap NodePool makes a fresh cluster unable to finish its first apply. Only cilium removes the taint; cilium arrives via ArgoCD; ArgoCD needs the cluster registered; registration needs a completed apply.dev-marcomandprod-marcomhit this and failed four times (run) —cert_managertimed out on its post-install hook after about 21 minutes, taking 10 downstream terragrunt units with it.Two changes break the cycle:
karpenter-default-nodepoolArgoCD app owns it and applies it fromcommon.yaml, so steady state is unchanged and the double-provisioning protection from feat(aws-eks-cluster): opt-in cilium startup taint on the default NodePool #895 is retained. The NodePool half was create-time only (ignore_changesonyaml_body), so existing clusters see no diff.karpenter_declare_cilium_startup_taintnow means only "this cluster will run cilium, so tolerate its startup taint." Variable description and README updated to match.Review focus
tolerationspluswebhook/cainjector/startupapicheck, against chart v1.20.2. Worth a second pair of eyes that these are the right keys and thatstartupapicheckhonours them on the hook Job.wait = truereleases (kubecost, jupyterhub) will deadlock a flag-on fresh cluster. Not addressed here.Test plan
terraform fmtis clean.terraform validatecannot pass standalone on this module — it errors on theaws.us-east-1provider alias that callers supply — and its output is byte-identical before and after this change, so it is uninformative rather than passing.The
lintcheck is red, inherited frommain: a Go typecheck failure inscripts/snowflake_generate_grant_allintroduced by 52fc695 (Snowflake provider bump) on 2026-09-01. Everymainrun before that was green. This PR touches no Go.Real verification is a fresh-cluster apply: re-run the marcom apply and confirm the
eksunit completes and the 10 skipped units run.