fix(pki): let a rotated host certificate re-record its own serial - #146
Merged
Conversation
Peer trust binds a live host row to the certificate serial recorded in
it. Nothing wrote that column when a host certificate was reissued:
RegisterHost errored on the mismatch and the daemon carried on, AdmitHost
refuses a live row outright, and no CLI sets it. On a cluster whose
certificates had been rotated, every daemon therefore refused every peer
with "replication RPC requires peer mTLS", replication stopped fleet-wide,
and the split-brain detector went blind for the same reason while the
workloads kept running and hid it. There was no way back in-product: the
correction has to reach the peer, and the stale serial is exactly what
blocks the peer channel — in both directions, since a pull is refused by
the same check.
RegisterHost now re-records the serial when a live row disagrees with the
certificate on disk. That caller is the one entitled to do it: the daemon
passes its own name and the serial it just read from its own PKI
directory, so the write only ever touches the node's own row, and anyone
able to change what that node presents already holds its private key. An
unreadable certificate ("unknown") never overwrites a good serial — a
local file-permission problem must not become a cluster-wide trust
downgrade. The removal tombstone is untouched and still routes through
AdmitHost, so a decommissioned host cannot register its way back in.
Self-recording converges a rotation on a healthy cluster, but cannot
rescue one that has already stopped replicating. For that there is
auth.trust_rotated_peer_certs, default false: set on every node it logs
and admits a mismatch for CA-issued HOST certificates instead of refusing,
long enough for the re-recorded serials to replicate, then it goes back
off with the pin enforcing against correct data. It relaxes the serial
comparison only — a removed host stays removed, and a distributable client
certificate still cannot act as a peer.
Both regressions observed red against the unfixed startup path, and the
recovery switch is asserted in both positions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Peer trust binds a live host row to the certificate serial recorded in it, and nothing wrote that column when a host certificate was reissued:
RegisterHosterrored on the mismatch and the daemon carried on,AdmitHostrefuses a live row outright, and no CLI sets it.A cluster whose host certificates had been rotated therefore had every daemon refuse every peer with
replication RPC requires peer mTLS. Replication stopped fleet-wide, and the split-brain detector went blind for the same reason while the workloads kept running and hid it. There was no way back in-product: the correction has to reach the peer, and the stale serial is exactly what blocks the peer channel — in both directions, since a pull is refused by the same check.Changes
RegisterHostre-records the serial when a live row disagrees with the certificate on disk. That caller is the one entitled to do it: the daemon passes its own name and the serial it just read from its own PKI directory, so the write only ever touches the node's own row, and anyone able to change what that node presents already holds its private key.unknown) never overwrites a good serial — a local file-permission problem must not become a cluster-wide trust downgrade.auth.trust_rotated_peer_certs, default false. Self-recording converges a rotation on a healthy cluster but cannot rescue one that has already stopped replicating; set on every node this logs and admits a mismatch for CA-issued host certificates instead of refusing, long enough for the re-recorded serials to replicate, then it goes back off with the pin enforcing against correct data.Security properties preserved
AdmitHost, so a decommissioned host cannot register its way back in — asserted with recovery mode on.Tests
Both startup regressions were observed red against the unfixed path, and the recovery switch is asserted in both positions. Previously nothing anywhere exercised a serial mismatch, which is why this shipped green.
Schema-neutral (v50 unchanged).
🤖 Generated with Claude Code