From 179f3d0d89a4e4a88489cacaec6dfeed7f495a17 Mon Sep 17 00:00:00 2001 From: Tamal Saha Date: Wed, 5 Aug 2026 11:09:01 +0600 Subject: [PATCH 1/5] postgres/remote-replica: TLS SAN guidance, custom port, new CLI flags Driven by a customer report: their replica looped on "Attempting pg_isready on primary" with SOURCE_SSL_MODE=verify-full while a non-TLS connection worked. Root cause is not a network or code problem: verify-full checks the dialed hostname against the server certificate's SANs, and the certificate is issued for in-cluster names only, so any external/load-balancer address fails the handshake. Adds a "TLS across clusters" section: verify-ca is the recommended mode between clusters; verify-full is supported guidance with the exact ReconfigureTLS ops request that adds the external hostname SAN to the server certificate (reissue and rotation handled by the ops manager, no manual restarts); and the pg_isready-loop symptom is named so the next person can recognize it. Updates the remote-config example for the new flags: -d host:port / --port (written into the AppBinding and honored end to end by the replica), --replica-name (emits a ready-to-apply replica manifest sized from the source spec), --auth-secret, plus notes on the fixed output path and the -y flag being a plain confirmation skip. Signed-off-by: Tamal Saha --- .../postgres/remote-replica/remotereplica.md | 68 ++++++++++++++++++- 1 file changed, 65 insertions(+), 3 deletions(-) diff --git a/docs/guides/postgres/remote-replica/remotereplica.md b/docs/guides/postgres/remote-replica/remotereplica.md index 5143a27bd4..27446166cd 100644 --- a/docs/guides/postgres/remote-replica/remotereplica.md +++ b/docs/guides/postgres/remote-replica/remotereplica.md @@ -199,12 +199,74 @@ pg-singapore nginx pg-singapore.something.org 172.104.37.147 80 ``` # Prepare for Remote Replica -We wil use the [kubedb_plugin](/docs/setup/README.md) for generating configuration for remote replica. It will create the appbinding and necessary secrets to connect with source server +We will use the [kubedb_plugin](/docs/setup/README.md) for generating configuration for remote replica. It creates the AppBinding and the secrets the replica needs to connect to the source server: + ```bash -$ kubectl dba remote-config postgres -n demo pg-singapore -uremote -ppass -d 172.104.37.147 -y -home/mehedi/go/src/kubedb.dev/yamls/postgres/pg-singapore-remote-config.yaml +$ kubectl dba remote-config postgres -n demo pg-singapore \ + -uremote -ppass \ + -d 172.104.37.147:5432 \ + --replica-name pg-london \ + -y +kubectl apply -f /home/user/pg-singapore-remote-config.yaml ``` +- `-d` takes the address the source is reachable on **from the replica's cluster** — a load + balancer frontend, not the in-cluster service. A non-standard port can be given as + `-d host:port` or with `--port` (it is written into the generated AppBinding's + `spec.clientConfig.service.port` and honored by the replica for the seed, streaming and + monitoring connections). +- `--replica-name` additionally emits a ready-to-apply remote replica `Postgres` manifest, + sized from the source's spec (version, replicas, storage, resources) with the + `remoteReplica` stanza and auth secret filled in. Treat it as a starting point — a + secondary site is often sized differently on purpose. +- `--auth-secret ` overrides the generated auth secret's name. +- The output file is always written to the **current directory** as + `-remote-config.yaml`; `-y` only skips the confirmation prompt. +- These flags require the kubectl-dba version shipped with this release or newer. + + +# TLS across clusters: sslmode and certificate SANs + +The generated AppBinding inherits its `sslmode` from the source database's `spec.sslMode`. +Two things matter when the replica connects through a load balancer or any external +endpoint: + +- **`verify-ca` is the recommended mode between clusters.** It verifies the server + certificate against your private CA (pinned via the exported `ca.crt`) without checking + the hostname — which is what you want when the same database is reached through + different names inside and outside its cluster. +- **`verify-full` additionally requires the exact hostname you dial to be present in the + server certificate's SAN list.** KubeDB issues the server certificate for the in-cluster + names, so a `verify-full` connection to an external address fails during the TLS + handshake unless that address was added to the certificate. The replica then loops + forever with `Attempting pg_isready on primary` while a plain (non-TLS) connection works + — that symptom almost always means a SAN mismatch, not a network problem. + +To use `verify-full`, add the external hostname to the source's server certificate. On an +existing database do it with a [ReconfigureTLS ops request](/docs/guides/postgres/reconfigure-tls/reconfigure-tls.md) +— certificates are reissued and rotated without manual restarts: + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: PostgresOpsRequest +metadata: + name: add-external-san + namespace: demo +spec: + type: ReconfigureTLS + databaseRef: + name: pg-singapore + tls: + certificates: + - alias: server + dnsNames: + - pg-singapore.example.com # the address the replica dials + apply: Always +``` + +Alternatively, edit the generated AppBinding's `spec.clientConfig.service.query` to +`sslmode=verify-ca`. + # Create Remote Replica We have prepared another cluster in london region for replicating across cluster. follow the installation instruction [above](/docs/README.md). From da3b7e028c2d52fb992e3f815b1ec9ed1b5af75a Mon Sep 17 00:00:00 2001 From: Tamal Saha Date: Wed, 5 Aug 2026 14:41:00 +0600 Subject: [PATCH 2/5] remote-replica: document the coordinator sidecar, lag monitor and standby Service Remote replica pods now run 2/2 with a pg-coordinator sidecar (no Raft) that self-heals on source timeline changes, keeps the standby role label truthful, and logs replication lag. A -standby Service exists at any replica count and is the correct entry point for read traffic; the primary-selecting Service has no endpoints while the database is a replica. All statements re-verified against the release stack in a full setup/failover/failback pass. Signed-off-by: Tamal Saha --- .../postgres/remote-replica/remotereplica.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/guides/postgres/remote-replica/remotereplica.md b/docs/guides/postgres/remote-replica/remotereplica.md index 27446166cd..6a95b78a0a 100644 --- a/docs/guides/postgres/remote-replica/remotereplica.md +++ b/docs/guides/postgres/remote-replica/remotereplica.md @@ -347,6 +347,21 @@ NAME VERSION STATUS AGE pg-london 18.3 Ready 7m17s ``` +Each remote replica pod runs `2/2` containers: `postgres` plus a `pg-coordinator` sidecar +in remote-replica mode (no Raft). The coordinator monitors streaming against the source, +recovers the replica with `pg_rewind` or a fresh basebackup when the source changes +timeline, keeps the pod's `standby` role label truthful (set only while the source +confirms the pod is streaming, cleared otherwise), and logs the replication lag: + +```bash +$ kubectl logs pg-london-0 -n demo -c pg-coordinator | grep LagMonitor +[LagMonitor] Pod pg-london-0: lag=0 B (in sync with source); next check in 40s +``` + +A `-standby` Service is created for remote replicas at any replica count and selects +the pods whose standby label is set — use it (not the primary-selecting `` Service, +which has no endpoints while the database is a replica) to route read-only traffic. + ## Validate Remote Replica At this point we want to validate the replication, we can see `pg-london-0` is connected as asynchronous replica From 1e49a493cc29f585f201102af07f71fc5fb694a3 Mon Sep 17 00:00:00 2001 From: Tamal Saha Date: Sat, 8 Aug 2026 16:21:55 +0600 Subject: [PATCH 3/5] postgres/remote-replica: add monitoring guide Documents the coordinator's DR metrics on the raft-metrics port, the spec.monitor + OnDelete pod-roll step, the NetworkPolicy scrape-allow needed on netpol-enabled clusters, ConfigMap-based dashboard provisioning (API imports do not survive a persistence-less Grafana restart), and how to read the dashboard. Written from a live setup on a public-IP two-cluster pair. Signed-off-by: Tamal Saha --- .../postgres/remote-replica/monitoring.md | 176 ++++++++++++++++++ 1 file changed, 176 insertions(+) create mode 100644 docs/guides/postgres/remote-replica/monitoring.md diff --git a/docs/guides/postgres/remote-replica/monitoring.md b/docs/guides/postgres/remote-replica/monitoring.md new file mode 100644 index 0000000000..24d4dc1ac0 --- /dev/null +++ b/docs/guides/postgres/remote-replica/monitoring.md @@ -0,0 +1,176 @@ +--- +title: Monitoring PostgreSQL Remote Replicas +menu: + docs_{{ .version }}: + identifier: pg-remote-replica-monitoring + name: Monitoring + parent: pg-remote-replica + weight: 30 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Monitoring PostgreSQL Remote Replicas + +A remote replica is disaster-recovery infrastructure: the questions its monitoring must +answer are *is the replica streaming*, *can it see its source*, *how much data is at risk +if the source data center is lost right now* (RPO), and *did self-healing fire*. This +guide wires those up on the **replica-side cluster** and installs a Grafana dashboard +built around exactly those questions. + +## What serves the metrics + +Every remote replica pod runs a `pg-coordinator` sidecar in remote-replica mode. Besides +recovery and role-label management, it serves DR metrics on the `raft-metrics` port +(23790), fed by its own monitor and lag-monitor loops — Prometheus scrapes never touch +the source database: + +| Metric | Meaning | +|---|---| +| `pg_coordinator_remote_replica_lag_bytes` | WAL bytes the source has written beyond what this pod has replayed — the data at risk (RPO). Absent until the first measurement | +| `pg_coordinator_remote_replica_streaming` | 1 when the source confirms this pod in `pg_stat_replication`; mirrors the pod's `standby` role label | +| `pg_coordinator_remote_replica_source_reachable` | 1 when the last source query succeeded — a DR replica that cannot see its source is not protecting anything | +| `pg_coordinator_remote_replica_last_lag_check_timestamp_seconds` | when the lag was last measured; the monitor backs off to 300s while in sync, so up to ~5 min of age is normal | +| `pg_coordinator_remote_replica_recovery_total{action,result}` | pg_rewind / pg_basebackup self-healing attempts; all four series exported from start, so any step above 0 is a real event | + +The standard `postgres_exporter` (port 56790, added by `spec.monitor`) contributes +`pg_replication_is_replica`, `pg_replication_lag_seconds`, `pg_stat_activity_count`, etc. + +## Prerequisites + +On the replica-side cluster: + +- [kube-prometheus-stack](https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack). + The dashboard below is developed against **Grafana 7.5.x** (`--set grafana.image.tag=7.5.5`). +- [Panopticon](https://appscode.com/products/panopticon/) with your KubeDB license — it + exports `kubedb_com_postgres_info`, which drives the dashboard's `app` variable. +- The `kubedb-metrics` chart (MetricsConfigurations for Panopticon). + +## Step 1: enable monitoring on the remote replica + +Add `spec.monitor` to the remote replica Postgres (the `release: prometheus` label must +match your kube-prometheus-stack release name — Prometheus only selects ServiceMonitors +carrying it): + +```yaml +spec: + monitor: + agent: prometheus.io/operator + prometheus: + serviceMonitor: + labels: + release: prometheus + interval: 30s +``` + +```bash +kubectl patch pg pg-london -n demo --type merge -p '{ + "spec": {"monitor": {"agent": "prometheus.io/operator", + "prometheus": {"serviceMonitor": {"labels": {"release": "prometheus"}, "interval": "30s"}}}}}' +``` + +This adds the exporter container to the pod template and creates the `-stats` +Service + ServiceMonitor exposing **both** metric ports (`metrics`/56790 and +`raft-metrics`/23790). + +> **The pod must be restarted once**: remote replica PetSets use the `OnDelete` update +> strategy, so the exporter container only appears after a pod delete. Streaming resumes +> automatically after the restart. + +```bash +kubectl delete pod pg-london-0 -n demo +kubectl wait pg pg-london -n demo --for=jsonpath='{.status.phase}'=Ready --timeout=300s +kubectl get pod pg-london-0 -n demo -o jsonpath='{range .spec.containers[*]}{.name} {end}' +# postgres pg-coordinator exporter +``` + +## Step 2: if the cluster runs NetworkPolicies, allow the scrape + +KubeDB can deploy NetworkPolicies that restrict ingress to database pods to their own +namespace (plus the operator). Prometheus lives in another namespace, so **both scrape +targets stay down** until you allow it. The symptom: `up{job="-stats"} == 0` while +`wget 127.0.0.1:56790/metrics` inside the pod works fine. + +```yaml +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-prometheus-scrape + namespace: demo +spec: + podSelector: + matchLabels: + app.kubernetes.io/component: database + app.kubernetes.io/managed-by: kubedb.com + policyTypes: + - Ingress + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: monitoring + ports: + - port: 56790 + protocol: TCP + - port: 23790 + protocol: TCP +``` + +Skip this step if `kubectl get networkpolicy -n demo` shows nothing. + +## Step 3: verify the scrape + +```bash +PROM=prometheus-prometheus-kube-prometheus-prometheus-0 +kubectl exec -n monitoring $PROM -c prometheus -- promtool query instant \ + http://localhost:9090 'up{namespace="demo",pod=~"pg-london-.*"}' +# both endpoints (metrics and raft-metrics) must be 1 + +kubectl exec -n monitoring $PROM -c prometheus -- promtool query instant \ + http://localhost:9090 'pg_coordinator_remote_replica_streaming{namespace="demo"}' +# 1 per streaming pod +``` + +## Step 4: install the dashboard + +The **KubeDB / Postgres / Remote Replica** dashboard +([opnpulse/dashboards, postgres folder](https://github.com/opnpulse/dashboards/tree/master/postgres)) +has three rows: *DR Protection Status* (streaming, source reachable, RPO in bytes, lag +data age, recoveries in 24h), *Replication Lag* (byte lag from the coordinator; apply +lag in seconds from the exporter — the latter also grows while the source is idle, read +them together), and *Self-Healing & Replica Health*. + +Provision it as a ConfigMap so it survives Grafana restarts (kube-prometheus-stack's +Grafana has no persistence — dashboards imported through the UI or API are lost on pod +restart; the sidecar re-provisions labeled ConfigMaps): + +```bash +kubectl create configmap pg-remote-replica-dashboard -n monitoring \ + --from-file=postgres_remote_replica_dashboard.json +kubectl label configmap pg-remote-replica-dashboard -n monitoring grafana_dashboard=1 +``` + +Then open Grafana and select your namespace and database in the `namespace` / `app` +variables: + +```bash +kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80 +``` + +## Reading the dashboard + +| Symptom | Likely meaning | +|---|---| +| Streaming red, Source Reachable green | a replica pod is down, diverged, or mid-recovery; watch Recovery Actions | +| Source Reachable red | the clusters are partitioned or the source is down — severity-1 even though the replica looks healthy locally | +| RPO climbing, Streaming green | WAL arrives but replay cannot keep up (or replay is paused) | +| Apply Lag climbing, byte lag 0 | the source is idle; not an incident | +| Recoveries ≥ 1 | self-healing fired (pg_rewind or re-seed) — read the coordinator logs of the affected pod | + +## Next Steps + +- [Remote Replica overview](/docs/guides/postgres/remote-replica/remotereplica.md) +- [Cross-Cluster DR with Bidirectional Failover](/docs/guides/postgres/remote-replica/advanced-setup.md) +- [Migration from Self-Managed PostgreSQL](/docs/guides/postgres/remote-replica/migration.md) From 9a4c31d3e653552902d53d15caa2f33744f3777f Mon Sep 17 00:00:00 2001 From: Tamal Saha Date: Tue, 11 Aug 2026 14:54:41 +0600 Subject: [PATCH 4/5] monitoring: warn against applying the scrape-allow policy on netpol-free clusters Whether KubeDB creates its NetworkPolicies is an install-time chart choice; on a cluster without them the scrape-allow policy becomes the only policy selecting the database pods and denies operator health checks, sticking the CR in Provisioning. Signed-off-by: Tamal Saha --- docs/guides/postgres/remote-replica/monitoring.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/guides/postgres/remote-replica/monitoring.md b/docs/guides/postgres/remote-replica/monitoring.md index 24d4dc1ac0..a9ff813f4e 100644 --- a/docs/guides/postgres/remote-replica/monitoring.md +++ b/docs/guides/postgres/remote-replica/monitoring.md @@ -118,7 +118,11 @@ spec: protocol: TCP ``` -Skip this step if `kubectl get networkpolicy -n demo` shows nothing. +Skip this step if `kubectl get networkpolicy -n demo` shows nothing — whether KubeDB +creates these policies is an install-time choice (`networkPolicy.enabled` in the chart +values). On a cluster **without** them, do **not** apply this policy: it would become the +only policy selecting the database pods and deny all other ingress — operator health +checks fail and the CR sticks in `Provisioning`. ## Step 3: verify the scrape From 620fcbbaf75eb8cb03d1ddb1c1a707531570d606 Mon Sep 17 00:00:00 2001 From: Tamal Saha Date: Tue, 18 Aug 2026 16:01:08 +0600 Subject: [PATCH 5/5] monitoring: fix menu weight collision with synchronous.md (30 -> 50) Signed-off-by: Tamal Saha --- docs/guides/postgres/remote-replica/monitoring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/postgres/remote-replica/monitoring.md b/docs/guides/postgres/remote-replica/monitoring.md index a9ff813f4e..fe127b0de6 100644 --- a/docs/guides/postgres/remote-replica/monitoring.md +++ b/docs/guides/postgres/remote-replica/monitoring.md @@ -5,7 +5,7 @@ menu: identifier: pg-remote-replica-monitoring name: Monitoring parent: pg-remote-replica - weight: 30 + weight: 50 menu_name: docs_{{ .version }} section_menu_id: guides ---