Skip to content

fix(pki): let a rotated host certificate re-record its own serial - #146

Merged
colonelpanik merged 1 commit into
mainfrom
fix/host-cert-rotation
Aug 8, 2026
Merged

fix(pki): let a rotated host certificate re-record its own serial#146
colonelpanik merged 1 commit into
mainfrom
fix/host-cert-rotation

Conversation

@colonelpanik

Copy link
Copy Markdown
Owner

Summary

Peer trust binds a live host row to the certificate serial recorded in it, and nothing wrote that column when a host certificate was reissued: RegisterHost errored on the mismatch and the daemon carried on, AdmitHost refuses a live row outright, and no CLI sets it.

A cluster whose host certificates had been rotated therefore had every daemon refuse every peer with replication RPC requires peer mTLS. Replication stopped fleet-wide, and the split-brain detector went blind for the same reason while the workloads kept running and hid it. There was no way back in-product: the correction has to reach the peer, and the stale serial is exactly what blocks the peer channel — in both directions, since a pull is refused by the same check.

Changes

  • RegisterHost re-records the serial when a live row disagrees with the certificate on disk. That caller is the one entitled to do it: the daemon passes its own name and the serial it just read from its own PKI directory, so the write only ever touches the node's own row, and anyone able to change what that node presents already holds its private key.
  • An unreadable certificate (unknown) never overwrites a good serial — a local file-permission problem must not become a cluster-wide trust downgrade.
  • New auth.trust_rotated_peer_certs, default false. Self-recording converges a rotation on a healthy cluster but cannot rescue one that has already stopped replicating; set on every node this logs and admits a mismatch for CA-issued host certificates instead of refusing, long enough for the re-recorded serials to replicate, then it goes back off with the pin enforcing against correct data.

Security properties preserved

  • The removal tombstone is untouched and still routes through AdmitHost, so a decommissioned host cannot register its way back in — asserted with recovery mode on.
  • Recovery mode relaxes the serial comparison only; a distributable client certificate still cannot act as a peer.
  • Revocation is unaffected — the CRL is signature-verified at the TLS handshake, before any of this is reached.

Tests

Both startup regressions were observed red against the unfixed path, and the recovery switch is asserted in both positions. Previously nothing anywhere exercised a serial mismatch, which is why this shipped green.

Schema-neutral (v50 unchanged).

🤖 Generated with Claude Code

Peer trust binds a live host row to the certificate serial recorded in
it. Nothing wrote that column when a host certificate was reissued:
RegisterHost errored on the mismatch and the daemon carried on, AdmitHost
refuses a live row outright, and no CLI sets it. On a cluster whose
certificates had been rotated, every daemon therefore refused every peer
with "replication RPC requires peer mTLS", replication stopped fleet-wide,
and the split-brain detector went blind for the same reason while the
workloads kept running and hid it. There was no way back in-product: the
correction has to reach the peer, and the stale serial is exactly what
blocks the peer channel — in both directions, since a pull is refused by
the same check.

RegisterHost now re-records the serial when a live row disagrees with the
certificate on disk. That caller is the one entitled to do it: the daemon
passes its own name and the serial it just read from its own PKI
directory, so the write only ever touches the node's own row, and anyone
able to change what that node presents already holds its private key. An
unreadable certificate ("unknown") never overwrites a good serial — a
local file-permission problem must not become a cluster-wide trust
downgrade. The removal tombstone is untouched and still routes through
AdmitHost, so a decommissioned host cannot register its way back in.

Self-recording converges a rotation on a healthy cluster, but cannot
rescue one that has already stopped replicating. For that there is
auth.trust_rotated_peer_certs, default false: set on every node it logs
and admits a mismatch for CA-issued HOST certificates instead of refusing,
long enough for the re-recorded serials to replicate, then it goes back
off with the pin enforcing against correct data. It relaxes the serial
comparison only — a removed host stays removed, and a distributable client
certificate still cannot act as a peer.

Both regressions observed red against the unfixed startup path, and the
recovery switch is asserted in both positions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@colonelpanik
colonelpanik merged commit 42f6641 into main Aug 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant