Skip to content

Add kubelet serving certificate readiness example - #365

Merged
kubernetes-prow[bot] merged 5 commits into
kubernetes-sigs:mainfrom
arnab-logs:docs/add-kubelet-serving-cert-readiness-example
Oct 1, 2026
Merged

kubernetes-prow[bot] merged 5 commits into
kubernetes-sigs:mainfrom
arnab-logs:docs/add-kubelet-serving-cert-readiness-example

Conversation

@arnab-logs

Copy link
Copy Markdown
Member

This PR adds a new example demonstrating how to use NRC to prevent workloads from scheduling on a node until kubelet has obtained its TLS serving certificate.

Includes:

  • examples/kubelet-cert-readiness/: kind cluster config and manifests (RBAC, NPD ConfigMaps, NPD DaemonSet, NodeReadinessRule)
  • docs/book/src/examples/kubelet-cert-readiness.md: mdBook documentation page
  • docs/book/src/SUMMARY.md: adds the new page to the site navigation

Use Case

This is in reference to a slack discussion raised in the NRC community. When a node is marked Ready, kubectl exec and kubectl logs may still fail because kubelet has not yet received its TLS serving certificate, the certificate is issued separately via a CSR and its approval can be delayed on some providers.

This causes visible failures in CI/CD environments: GitLab Runner marks jobs as failed when it cannot exec into pods on a new node, and Fluentbit fails to collect logs from kubelet before the certificate is in place.

How to test

Create kind cluster

kind create cluster --config examples/kubelet-cert-readiness/kind-config.yaml

Install the CRDs and Controller

See the Installation Guide for details

Deploy the Example

kubectl apply -f examples/kubelet-cert-readiness/manifests/

Check the startup taint is applied

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
Pasted Graphic

Check the node condition

   kubectl get node <node-name> \
    -o jsonpath='{.status.conditions[?(@.type=="KubeletServingCertNotReady")]}' | jq .
Pasted Graphic 1

Approve the worker's pending CSR to simulate the certificate being issued

    kubectl get csr
    # find the entry with SIGNERNAME kubernetes.io/kubelet-serving and REQUESTOR system:node:<worker-name>
    kubectl certificate approve <worker-csr-name>
Pasted Graphic 3

Check the condition again

    kubectl get node <node-name> \
    -o jsonpath='{.status.conditions[?(@.type=="KubeletServingCertNotReady")]}' | jq .
Pasted Graphic 4

Check taint removal

   kubectl get node <node-name> -o jsonpath='{.spec.taints}'
Pasted Graphic 5

Confirm kubectl exec works

   kubectl run test-pod --image=busybox --restart=Never -- sleep 3600
   
   kubectl exec test-pod -- echo "exec works"
image

@netlify

netlify Bot commented Aug 7, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for node-readiness-controller canceled.

Name Link
🔨 Latest commit 310d423
🔍 Latest deploy log https://app.netlify.com/projects/node-readiness-controller/deploys/6abe5d0bc690f50008d90d74

@kubernetes-prow kubernetes-prow Bot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 7, 2026
@arnab-logs

Copy link
Copy Markdown
Member Author

Looks like the prow job failed on the docs link check. I verified the files are present in this PR and the failure seems to be because the links point to main where these new files don't exist yet

@ajaysundark ajaysundark left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@kubernetes-prow kubernetes-prow Bot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Sep 19, 2026
@Karthik-K-N

Copy link
Copy Markdown
Contributor

@arnab-logs could you please update the PR and fix the failing test. Thank you.

…ntation

Updated links to fix prow failures and point to the specific commit for kubelet-cert-readiness manifests and kind-config.yaml.
@kubernetes-prow kubernetes-prow Bot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 27, 2026
Updated links in kubelet-cert-readiness.md to point to the latest commit.
@arnab-logs

Copy link
Copy Markdown
Member Author

@Karthik-K-N Fixed the failing test, PR is ready to be merged. PTAL!

else
echo "kubelet serving certificate not yet present at $CERT_PATH"
exit 1
fi No newline at end of file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add the missing new lines to all the manifest files.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made the necessary changes needed here.

@kubernetes-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ajaysundark, arnab-logs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow kubernetes-prow Bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 1, 2026
@arnab-logs
arnab-logs force-pushed the docs/add-kubelet-serving-cert-readiness-example branch from 7785374 to d123bf2 Compare October 1, 2026 12:51
@kubernetes-prow kubernetes-prow Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 1, 2026
@arnab-logs
arnab-logs force-pushed the docs/add-kubelet-serving-cert-readiness-example branch from d123bf2 to 310d423 Compare October 1, 2026 13:15

@Karthik-K-N Karthik-K-N left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you
/lgtm

If possible please squash the commits

@kubernetes-prow kubernetes-prow Bot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 1, 2026
@kubernetes-prow
kubernetes-prow Bot merged commit c45291b into kubernetes-sigs:main Oct 1, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants