Cascade a multi-replica remote replica, and drop the inherited auto.conf - #70
Merged
Conversation
Two independent problems in the remote-replica seed path, both hit during a
customer migration from a repmgr-managed source.
1. postgresql.auto.conf was inherited from the source.
pg_basebackup copies $PGDATA/postgresql.auto.conf verbatim, and Postgres reads
that file AFTER postgresql.conf -- so every setting the source had ALTER
SYSTEM'd silently overrode the recovery configuration this script had just
written. On a repmgr- or patroni-managed source that includes primary_conninfo
and primary_slot_name, so the replica demanded a slot belonging to the manager's
own standby:
FATAL: could not start WAL streaming: ERROR: replication slot
"repmgr_slot_1" does not exist
repeating until the source recycled the WAL the seed still needed, at which
point the base backup was dead and had to be retaken. Nothing in this repo ever
touched that file.
We pass no -R to pg_basebackup, so nothing we depend on lives in it. Truncate it
once, at seed time, inside the fresh-PGDATA branch -- not on every start, which
would discard a user's own later ALTER SYSTEM. The previous contents are echoed
first, so an inherited setting leaves an audit line instead of vanishing.
2. Every replica seeded and streamed from the external source.
All pods ran this script against PRIMARY_HOST, so a 3-replica remote replica
took three base backups and held three WAL streams across the WAN against the
customer's production database.
Now only ordinal 0 follows the external source; every other pod follows ordinal
0. One base backup, one stream. Ordinal 0's catalog is a byte copy of the
source's, so the operator-supplied credentials authenticate against it too, and
peer connections present our own certs rather than the source's.
The topology is also the one wanted after cutover: promoting ordinal 0 turns the
existing cascade into an ordinary primary-with-standbys tree, with no re-seed of
the followers. Verified: after promotion both followers stayed on zero base
backups and simply followed the timeline switch to 2.
GOVERNING_SERVICE_DNS supplies ordinal 0's address; the fallback derives it from
HOSTNAME and NAMESPACE so the script still works against an operator that does
not set it.
Signed-off-by: souravbiswassanto <saurov@appscode.com>
The truncation added in the previous commit ran only inside the fresh-PGDATA branch, which is not the only way a foreign postgresql.auto.conf reaches $PGDATA. The coordinator's own re-seed (pg_basebackup in pg_basebackup.go) and pg_rewind both copy from the upstream and both leave PGDATA populated, so the seed block -- and with it the truncation -- is skipped on the restart that follows. The upstream's primary_conninfo and primary_slot_name were then back in force, silently overriding what this script writes into postgresql.conf, because Postgres reads auto.conf second. That is a loop, not just a missed cleanup: streaming fails for a slot that was never ours, the coordinator sees the pod absent from the upstream's pg_stat_replication, re-seeds, and the re-seed restores the same auto.conf. Run the sanitisation unconditionally instead, and make it selective rather than truncating: strip only the keys KubeDB authors itself (primary_conninfo, primary_slot_name, restore_command, recovery_target*, recovery_min_apply_delay, archive_mode/command/library) and leave everything else alone. That is safe to run on every start -- a user's own ALTER SYSTEM survives -- so it no longer depends on the fresh-seed guard, and it covers every path that can seed rather than just the one. Passwords in the reported primary_conninfo are redacted in the log line. Signed-off-by: souravbiswassanto <saurov@appscode.com>
The cascade leg connected to ordinal 0 as the local POSTGRES_USER, and the
pg_hba.conf this script generates names the literal user "postgres" in every
replication rule. Two problems follow, and neither shows up in a lab where
the client happens to hand over their superuser.
First, the dependency is on the wrong role. Ordinal 0's pg_authid is a byte
copy of the source's, so the role that definitionally exists there with
REPLICATION is the one the source DBA created for us to base-backup with --
PRIMARY_USER_NAME. "postgres" carries no such guarantee: a client who hands
over a restricted user instead of their superuser may have no postgres role
at all, which is why the migration runbook already tells them to check for it
and CREATE ROLE postgres if it is missing. The cascade should not depend on
that remediation having happened.
Second, `all` in pg_hba's DATABASE column does not match a replication
connection, so once the connecting user is anything other than "postgres"
there is no matching rule and the peer is refused before authentication:
FATAL: no pg_hba.conf entry for replication connection
from host "10.42.0.230", user "migrator", no encryption
Use PRIMARY_USER_NAME for the cascade leg, and clone each "postgres"
replication rule for that user rather than re-deriving the auth method per
CLIENT_AUTH_MODE, so the two sets cannot drift.
cert mode keeps the local user: clientcert=verify-full binds the certificate
CN to the role name, and our client certificate is issued for this cluster's
own user, so that is the only role it can present as.
Verified live on a three-replica cascade whose source user is 'migrator':
pg_hba gains the two cloned rules with the scram method preserved, both
followers' primary_conninfo carries user=migrator, pg_stat_replication shows
all three streaming as migrator, and the physical replication connection that
previously returned the FATAL above now succeeds.
Signed-off-by: souravbiswassanto <saurov@appscode.com>
sanitize_auto_conf ran `grep -Ev "$pattern" "$f"` unguarded under
`set -eou pipefail`. grep exits 1 when it selects no lines, which happens
whenever the inherited postgresql.auto.conf consists *only* of keys we
strip -- exactly what an upstream managed entirely by repmgr or patroni
looks like, since primary_conninfo and primary_slot_name may be the whole
file.
The failure is silent and total. The role script dies immediately after
logging what it was about to strip, so the file is left unmodified with the
foreign primary_conninfo still in it; postgres never starts; the supervisor
notices the server is down and re-runs the role script, which dies at the
same line, forever. Meanwhile both containers report Ready with zero
restarts, so nothing at the pod level indicates a problem -- only the
Postgres CR going Critical does.
Observed on a remote replica whose postgresql.auto.conf held only
primary_conninfo and primary_slot_name:
stripping inherited recovery settings from postgresql.auto.conf:
| primary_conninfo = 'host=old-manager user=repmgr'
| primary_slot_name = 'repmgr_slot_1'
removing the initial scripts as server is not running ...
running the initial script ...
Running as Remote Replica
... same three lines again, 4 times in 2 minutes
A PostgreSQL-written auto.conf normally carries ALTER SYSTEM's two comment
lines, which never match the pattern and so keep grep at exit 0 -- that is
why the common path survives. Any upstream presenting a headerless file
hits this.
Treat exit 1 as the normal "nothing left to keep" outcome and write the
resulting empty file. Only exit >1 is a real grep failure, and there the
original is left untouched rather than replaced by a truncated temp file.
Signed-off-by: Tamal Saha <tamal@appscode.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent problems in the remote-replica seed path, both hit during a
customer migration from a repmgr-managed source.
pg_basebackup copies $PGDATA/postgresql.auto.conf verbatim, and Postgres reads
that file AFTER postgresql.conf -- so every setting the source had ALTER
SYSTEM'd silently overrode the recovery configuration this script had just
written. On a repmgr- or patroni-managed source that includes primary_conninfo
and primary_slot_name, so the replica demanded a slot belonging to the manager's
own standby:
FATAL: could not start WAL streaming: ERROR: replication slot
"repmgr_slot_1" does not exist
repeating until the source recycled the WAL the seed still needed, at which
point the base backup was dead and had to be retaken. Nothing in this repo ever
touched that file.
We pass no -R to pg_basebackup, so nothing we depend on lives in it. Truncate it
once, at seed time, inside the fresh-PGDATA branch -- not on every start, which
would discard a user's own later ALTER SYSTEM. The previous contents are echoed
first, so an inherited setting leaves an audit line instead of vanishing.
All pods ran this script against PRIMARY_HOST, so a 3-replica remote replica
took three base backups and held three WAL streams across the WAN against the
customer's production database.
Now only ordinal 0 follows the external source; every other pod follows ordinal
0. One base backup, one stream. Ordinal 0's catalog is a byte copy of the
source's, so the operator-supplied credentials authenticate against it too, and
peer connections present our own certs rather than the source's.
The topology is also the one wanted after cutover: promoting ordinal 0 turns the
existing cascade into an ordinary primary-with-standbys tree, with no re-seed of
the followers. Verified: after promotion both followers stayed on zero base
backups and simply followed the timeline switch to 2.
GOVERNING_SERVICE_DNS supplies ordinal 0's address; the fallback derives it from
HOSTNAME and NAMESPACE so the script still works against an operator that does
not set it.
Signed-off-by: souravbiswassanto saurov@appscode.com