postgres/remote-replica: TLS SAN guidance, custom port, new CLI flags #1040
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,180 @@ | ||
| --- | ||
| title: Monitoring PostgreSQL Remote Replicas | ||
| menu: | ||
| docs_{{ .version }}: | ||
| identifier: pg-remote-replica-monitoring | ||
| name: Monitoring | ||
| parent: pg-remote-replica | ||
| weight: 50 | ||
| menu_name: docs_{{ .version }} | ||
| section_menu_id: guides | ||
| --- | ||
|
|
||
| > New to KubeDB? Please start [here](/docs/README.md). | ||
|
|
||
| # Monitoring PostgreSQL Remote Replicas | ||
|
|
||
| A remote replica is disaster-recovery infrastructure: the questions its monitoring must | ||
| answer are *is the replica streaming*, *can it see its source*, *how much data is at risk | ||
| if the source data center is lost right now* (RPO), and *did self-healing fire*. This | ||
| guide wires those up on the **replica-side cluster** and installs a Grafana dashboard | ||
| built around exactly those questions. | ||
|
|
||
| ## What serves the metrics | ||
|
|
||
| Every remote replica pod runs a `pg-coordinator` sidecar in remote-replica mode. Besides | ||
| recovery and role-label management, it serves DR metrics on the `raft-metrics` port | ||
| (23790), fed by its own monitor and lag-monitor loops — Prometheus scrapes never touch | ||
| the source database: | ||
|
|
||
| | Metric | Meaning | | ||
| |---|---| | ||
| | `pg_coordinator_remote_replica_lag_bytes` | WAL bytes the source has written beyond what this pod has replayed — the data at risk (RPO). Absent until the first measurement | | ||
| | `pg_coordinator_remote_replica_streaming` | 1 when the source confirms this pod in `pg_stat_replication`; mirrors the pod's `standby` role label | | ||
| | `pg_coordinator_remote_replica_source_reachable` | 1 when the last source query succeeded — a DR replica that cannot see its source is not protecting anything | | ||
| | `pg_coordinator_remote_replica_last_lag_check_timestamp_seconds` | when the lag was last measured; the monitor backs off to 300s while in sync, so up to ~5 min of age is normal | | ||
| | `pg_coordinator_remote_replica_recovery_total{action,result}` | pg_rewind / pg_basebackup self-healing attempts; all four series exported from start, so any step above 0 is a real event | | ||
|
|
||
| The standard `postgres_exporter` (port 56790, added by `spec.monitor`) contributes | ||
| `pg_replication_is_replica`, `pg_replication_lag_seconds`, `pg_stat_activity_count`, etc. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| On the replica-side cluster: | ||
|
|
||
| - [kube-prometheus-stack](https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack). | ||
| The dashboard below is developed against **Grafana 7.5.x** (`--set grafana.image.tag=7.5.5`). | ||
| - [Panopticon](https://appscode.com/products/panopticon/) with your KubeDB license — it | ||
| exports `kubedb_com_postgres_info`, which drives the dashboard's `app` variable. | ||
| - The `kubedb-metrics` chart (MetricsConfigurations for Panopticon). | ||
|
|
||
| ## Step 1: enable monitoring on the remote replica | ||
|
|
||
| Add `spec.monitor` to the remote replica Postgres (the `release: prometheus` label must | ||
| match your kube-prometheus-stack release name — Prometheus only selects ServiceMonitors | ||
| carrying it): | ||
|
|
||
| ```yaml | ||
| spec: | ||
| monitor: | ||
| agent: prometheus.io/operator | ||
| prometheus: | ||
| serviceMonitor: | ||
| labels: | ||
| release: prometheus | ||
| interval: 30s | ||
| ``` | ||
|
|
||
| ```bash | ||
| kubectl patch pg pg-london -n demo --type merge -p '{ | ||
| "spec": {"monitor": {"agent": "prometheus.io/operator", | ||
| "prometheus": {"serviceMonitor": {"labels": {"release": "prometheus"}, "interval": "30s"}}}}}' | ||
| ``` | ||
|
|
||
| This adds the exporter container to the pod template and creates the `<db>-stats` | ||
| Service + ServiceMonitor exposing **both** metric ports (`metrics`/56790 and | ||
| `raft-metrics`/23790). | ||
|
|
||
| > **The pod must be restarted once**: remote replica PetSets use the `OnDelete` update | ||
| > strategy, so the exporter container only appears after a pod delete. Streaming resumes | ||
| > automatically after the restart. | ||
|
|
||
| ```bash | ||
| kubectl delete pod pg-london-0 -n demo | ||
| kubectl wait pg pg-london -n demo --for=jsonpath='{.status.phase}'=Ready --timeout=300s | ||
| kubectl get pod pg-london-0 -n demo -o jsonpath='{range .spec.containers[*]}{.name} {end}' | ||
| # postgres pg-coordinator exporter | ||
| ``` | ||
|
|
||
| ## Step 2: if the cluster runs NetworkPolicies, allow the scrape | ||
|
|
||
| KubeDB can deploy NetworkPolicies that restrict ingress to database pods to their own | ||
| namespace (plus the operator). Prometheus lives in another namespace, so **both scrape | ||
| targets stay down** until you allow it. The symptom: `up{job="<db>-stats"} == 0` while | ||
| `wget 127.0.0.1:56790/metrics` inside the pod works fine. | ||
|
|
||
| ```yaml | ||
| apiVersion: networking.k8s.io/v1 | ||
| kind: NetworkPolicy | ||
| metadata: | ||
| name: allow-prometheus-scrape | ||
| namespace: demo | ||
| spec: | ||
| podSelector: | ||
| matchLabels: | ||
| app.kubernetes.io/component: database | ||
| app.kubernetes.io/managed-by: kubedb.com | ||
| policyTypes: | ||
| - Ingress | ||
| ingress: | ||
| - from: | ||
| - namespaceSelector: | ||
| matchLabels: | ||
| kubernetes.io/metadata.name: monitoring | ||
| ports: | ||
| - port: 56790 | ||
| protocol: TCP | ||
| - port: 23790 | ||
| protocol: TCP | ||
| ``` | ||
|
|
||
| Skip this step if `kubectl get networkpolicy -n demo` shows nothing — whether KubeDB | ||
| creates these policies is an install-time choice (`networkPolicy.enabled` in the chart | ||
| values). On a cluster **without** them, do **not** apply this policy: it would become the | ||
| only policy selecting the database pods and deny all other ingress — operator health | ||
| checks fail and the CR sticks in `Provisioning`. | ||
|
|
||
| ## Step 3: verify the scrape | ||
|
|
||
| ```bash | ||
| PROM=prometheus-prometheus-kube-prometheus-prometheus-0 | ||
| kubectl exec -n monitoring $PROM -c prometheus -- promtool query instant \ | ||
| http://localhost:9090 'up{namespace="demo",pod=~"pg-london-.*"}' | ||
| # both endpoints (metrics and raft-metrics) must be 1 | ||
|
|
||
| kubectl exec -n monitoring $PROM -c prometheus -- promtool query instant \ | ||
| http://localhost:9090 'pg_coordinator_remote_replica_streaming{namespace="demo"}' | ||
| # 1 per streaming pod | ||
| ``` | ||
|
|
||
| ## Step 4: install the dashboard | ||
|
|
||
| The **KubeDB / Postgres / Remote Replica** dashboard | ||
| ([opnpulse/dashboards, postgres folder](https://github.com/opnpulse/dashboards/tree/master/postgres)) | ||
| has three rows: *DR Protection Status* (streaming, source reachable, RPO in bytes, lag | ||
| data age, recoveries in 24h), *Replication Lag* (byte lag from the coordinator; apply | ||
| lag in seconds from the exporter — the latter also grows while the source is idle, read | ||
| them together), and *Self-Healing & Replica Health*. | ||
|
|
||
| Provision it as a ConfigMap so it survives Grafana restarts (kube-prometheus-stack's | ||
| Grafana has no persistence — dashboards imported through the UI or API are lost on pod | ||
| restart; the sidecar re-provisions labeled ConfigMaps): | ||
|
|
||
| ```bash | ||
| kubectl create configmap pg-remote-replica-dashboard -n monitoring \ | ||
| --from-file=postgres_remote_replica_dashboard.json | ||
| kubectl label configmap pg-remote-replica-dashboard -n monitoring grafana_dashboard=1 | ||
|
Comment on lines
+142
to
+156
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== target file around affected lines =="
if [ -f docs/guides/postgres/remote-replica/monitoring.md ]; then
sed -n '35,60p;130,160p' docs/guides/postgres/remote-replica/monitoring.md | cat -n
else
echo "target file not found"
fi
echo
echo "== dashboard json references in target file =="
rg -n "postgres_remote_replica_dashboard|external|dashboard|Grafana|7\.5|7\.4|github.com/opnpulse/dashboards" docs/guides/postgres/remote-replica/monitoring.md || true
echo
echo "== check public file listing and README version for referenced path =="
url="https://api.github.com/repos/opnpulse/dashboards/git/trees/master?recursive=1"
python3 - <<'PY'
import json, urllib.request
from urllib.request import Request, HTTPError
url="https://api.github.com/repos/opnpulse/dashboards/git/trees/master?recursive=1"
req=Request(url, headers={'User-Agent':'CodeRabbit'})
try:
with urllib.request.urlopen(req, timeout=20) as r:
data=json.load(r)
except Exception as e:
print("GITHUB_TREE_ERROR:", type(e).__name__, e)
raise SystemExit(0)
target="postgres_remote_replica_dashboard.json"
matches=[]
paths=[]
for item in data.get("tree", []):
paths.append(item["path"])
if item["path"] == target:
matches.append(item)
print("master_contains_target_file:", len(matches)>0, matches[:5])
postgres_files=sorted(p for p in paths if p.startswith("postgres/") and p.endswith(".json"))
print("postgres_json_files=", postgres_files)
# find README line possibly mentioning versions
readme="".join(data.get("tree", [])[0]) if any(p=="README.md" for p in paths) else None
</PY
python3 - <<'PY'
import urllib.request
url="https://raw.githubusercontent.com/opnpulse/dashboards/master/README.md"
print("README fetch:", end=" ")
try:
text=urllib.request.urlopen(url, timeout=20).read().decode()
print("ok")
for i,line in enumerate(text.splitlines(),1):
if "Grafana" in line or "7.4" in line or "7.5" in line:
print(f"{i}: {line[:200]}")
except Exception as e:
print("error:", e)
PYRepository: kubedb/docs Length of output: 4426 🏁 Script executed: #!/bin/bash
set -euo pipefail
python3 - <<'PY'
import urllib.request
import json
url = "https://api.github.com/repos/opnpulse/dashboards/git/trees/master?recursive=1"
req = urllib.request.Request(url, headers={"User-Agent": "CodeRabbit"})
try:
with urllib.request.urlopen(req, timeout=20) as r:
data = json.load(r)
except Exception as e:
print("GITHUB_TREE_ERROR:", type(e).__name__, e)
raise SystemExit(0)
target = "postgres_remote_replica_dashboard.json"
paths = [item["path"] for item in data.get("tree", [])]
print("master_contains_target_file:", target in paths)
print("matched_files:")
for item in data.get("tree", []):
if item["path"] == target:
print(item)
print("\npostgres_json_files:")
for p in sorted(p for p in paths if p.startswith("postgres/") and p.endswith(".json")):
print(p)
readme_content = None
for item in data.get("tree", []):
if item["path"] == "README.md" and item["type"] == "blob":
with urllib.request.urlopen(item["url"], timeout=20) as r:
readme_content = r.read().decode(errors="replace")
break
print("\nGrafana/version mentions in README.md:")
if readme_content:
for i, line in enumerate(readme_content.splitlines(), 1):
if "Grafana" in line or "7.4" in line or "7.5" in line:
print(f"{i}: {line}")
else:
print("README.md not found")
PYRepository: kubedb/docs Length of output: 312 🌐 Web query:
💡 Result: The requested file, postgres_remote_replica_dashboard.json, does not appear to exist within the public repositories of the OpenPulse (opnpulse) GitHub organization [1]. While OpenPulse is an OpenTelemetry-based observability project maintained by AppsCode [1][2], public repositories associated with it, such as opnpulse/installer [3] and opnpulse/alerts [4], focus on infrastructure components, Helm charts [5][6], and runbooks [7], rather than providing a library of specific Grafana dashboard JSON files for PostgreSQL. It is possible the file you are looking for originated in a different repository, such as the AppsCode grafana-dashboards repository, which does contain various PostgreSQL-related dashboards [8][9][10], or it may be a custom or non-public file. For PostgreSQL observability with Prometheus and Grafana, the standard community source for such dashboards is the prometheus-community/postgres_exporter repository [11], which provides officially maintained dashboard templates. Citations:
Use a dashboard source that contains
📍 Affects 1 file
🤖 Prompt for AI Agents |
||
| ``` | ||
|
|
||
| Then open Grafana and select your namespace and database in the `namespace` / `app` | ||
| variables: | ||
|
|
||
| ```bash | ||
| kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80 | ||
| ``` | ||
|
|
||
| ## Reading the dashboard | ||
|
|
||
| | Symptom | Likely meaning | | ||
| |---|---| | ||
| | Streaming red, Source Reachable green | a replica pod is down, diverged, or mid-recovery; watch Recovery Actions | | ||
| | Source Reachable red | the clusters are partitioned or the source is down — severity-1 even though the replica looks healthy locally | | ||
| | RPO climbing, Streaming green | WAL arrives but replay cannot keep up (or replay is paused) | | ||
| | Apply Lag climbing, byte lag 0 | the source is idle; not an incident | | ||
| | Recoveries ≥ 1 | self-healing fired (pg_rewind or re-seed) — read the coordinator logs of the affected pod | | ||
|
|
||
| ## Next Steps | ||
|
|
||
| - [Remote Replica overview](/docs/guides/postgres/remote-replica/remotereplica.md) | ||
| - [Cross-Cluster DR with Bidirectional Failover](/docs/guides/postgres/remote-replica/advanced-setup.md) | ||
| - [Migration from Self-Managed PostgreSQL](/docs/guides/postgres/remote-replica/migration.md) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Use descriptive link text.
Replace
[here]with text that identifies the destination, such as[the KubeDB documentation overview]. This improves link context and resolves MD059.🧰 Tools
🪛 markdownlint-cli2 (0.23.2)
[warning] 13-13: Link text should be descriptive
(MD059, descriptive-link-text)
🤖 Prompt for AI Agents
Source: Linters/SAST tools