From 3bd12b61837bde4e337094d7cff6318db7fda492 Mon Sep 17 00:00:00 2001 From: Tamal Saha Date: Wed, 1 Jul 2026 07:46:26 +0600 Subject: [PATCH] Add Redis cross-DC disaster recovery (DC-DR) docs Document DC-DR for Redis/Valkey under docs/guides/redis/dr/: an overview (concepts, 2-DC vs 3-DC topologies, deploy, observe status.disasterRecovery, unplanned failover, planned near-zero-RPO switchover, failback, scaling, cleanup) and a scenario runbook. Native scope is Sentinel and Standalone; the cross-DC link is a plain async REPLICAOF, the writable DC is chosen by the dr-controlplane primary-DC Lease, and the rd-coordinator fence holds a non-active DC's master standby, fail closed. Signed-off-by: Tamal Saha --- docs/guides/redis/dr/_index.md | 10 + docs/guides/redis/dr/overview/index.md | 362 +++++++++++++++++++++++++ docs/guides/redis/dr/runbook/index.md | 256 +++++++++++++++++ 3 files changed, 628 insertions(+) create mode 100644 docs/guides/redis/dr/_index.md create mode 100644 docs/guides/redis/dr/overview/index.md create mode 100644 docs/guides/redis/dr/runbook/index.md diff --git a/docs/guides/redis/dr/_index.md b/docs/guides/redis/dr/_index.md new file mode 100644 index 0000000000..6f0435319d --- /dev/null +++ b/docs/guides/redis/dr/_index.md @@ -0,0 +1,10 @@ +--- +title: Disaster Recovery +menu: + docs_{{ .version }}: + identifier: rd-dr + name: DR + parent: rd-redis-guides + weight: 95 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/redis/dr/overview/index.md b/docs/guides/redis/dr/overview/index.md new file mode 100644 index 0000000000..ff1513ca83 --- /dev/null +++ b/docs/guides/redis/dr/overview/index.md @@ -0,0 +1,362 @@ +--- +title: DC-DR Overview +menu: + docs_{{ .version }}: + identifier: rd-dr-overview + name: Overview + parent: rd-dr + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +# Cross Data Center Disaster Recovery (DC-DR) for Redis + +KubeDB can run a single distributed `Redis` (or `Valkey`, a drop-in Redis fork with the +identical replication mechanics) across multiple data centers so the database survives the +loss of an entire data center (DC). Exactly one DC is writable at any instant; the other is a +warm, read-only standby that streams from it across the DCs. When the active DC is lost, +KubeDB promotes the surviving DC, and the single connection endpoint follows the new writable +DC. + +DC-DR adapts the Postgres/MariaDB DC-DR design to Redis. It makes the `dr-controlplane` Lease +the authority that decides which DC is writable, adds a fail-closed fence in the +`rd-coordinator`, wires the cross-DC async link with plain `REPLICAOF`, and presents one CR +over the per-DC Redis deployments. + +> **Native scope: Sentinel and Standalone.** Only Sentinel and Standalone modes have a native +> cross-cluster replication primitive (plain `REPLICAOF` between two independent +> deployments), so those are the initial DC-DR scope. **Cluster mode** has no native +> cross-cluster replication (a `cluster-enabled` node rejects `REPLICAOF` to a node outside +> its own gossip ring, and stretching one gossip ring across DCs is the forbidden +> anti-pattern), so cross-DC DR for Cluster mode needs an external logical-sync tool +> (RedisShake-style) and is out of the initial native scope. + +This page is the conceptual overview and a quick start. See also: + +- [DC-DR Runbook](/docs/guides/redis/dr/runbook/index.md) for scenario-by-scenario + procedures. + +> **New to KubeDB?** Please start [here](/docs/README.md). + +## How it works + +DC-DR is built on one rule: **the gossip ring or Sentinel quorum never stretches across data +centers; a plain cross-DC `REPLICAOF` link is the only thing that crosses the DC boundary.** + +- **Each data center is a self-contained Redis.** The operator expands the single `Redis` CR + into one Redis per data-bearing DC. In Sentinel mode each DC runs its own master, replicas, + and Sentinel quorum for intra-DC HA; in Standalone mode each DC runs its own master (with + optional replicas). The Sentinel quorum (or the intra-DC replication topology) never crosses + the DC boundary, so cross-DC latency or a partition cannot flap an intra-DC election. +- **One cross-DC authority decides who is writable.** A small control plane + (`dr-controlplane`), backed by a three-site etcd quorum, publishes one `coordination.k8s.io` + **Lease** per failover scope. The DC that holds the Lease is the **active** (writable) DC, + and its master is the one writable master. This is the single cross-DC failover decision. +- **Cross-DC replication is a plain async `REPLICAOF` (net-new).** The standby DC's master is + `REPLICAOF` the active DC's master, a chained async replica that streams the replication + backlog cross-DC. With more than two data DCs, the active master feeds one `REPLICAOF` + replica per standby DC. An intra-active-DC failover (Sentinel electing a new master within + the active DC) is transparent to the standby, because the standby points at the active DC's + role-pinned primary `Service`, which follows whichever pod is the active master. +- **Writability is fenced locally and fails closed.** A per-DC `dr-controlplane` agent projects + the Lease holder onto its own spoke cluster as a small marker `ConfigMap` (`primary-dc` in + the `dc-failover` namespace, `data.activeDC` + `data.renewTime`, 30s TTL). The + `rd-coordinator` fence reads only that local marker: if it cannot confirm its DC holds the + Lease (the DC lost it, is partitioned, or the marker is missing or stale), it keeps the + local master labeled `standby` and never promotes it. Because the fence lives in the DC and + fails closed, a cut-off old-active DC stops advertising as primary on its own, before the + hub even reacts. This local fence plus the etcd majority (only one DC can hold the Lease) is + the split-brain guarantee. +- **Only the active DC's master is labeled `primary`.** Each DC's Sentinel elects its own + local master, but a non-active DC's master is held `kubedb.com/role: standby` by the fence, + so the single primary `Service` and the `AppBinding` always resolve to the active DC's + writable master. Standby DCs keep `replica-read-only yes`. + +### Data center roles + +Each DC plays one role, set on the `PlacementPolicy` `distributionRule.role`: + +| Role | Holds Redis data | Primary eligible | Purpose | +| --- | --- | --- | --- | +| **Member** | yes | yes | A self-contained Redis; a candidate for the active (writable) DC. | +| **Arbiter** | no | no | Vote only, the `dr-controlplane` etcd tie-breaker; runs no Redis. | + +> Redis needs no cross-DC voter of its own: its quorum (gossip or Sentinel) is intra-DC. The +> Arbiter DC holds only the `dr-controlplane` etcd member and no Redis. + +A typical layout is two Member DCs plus one vote-only Arbiter DC: the three-site etcd quorum +lives across all three sites, but Redis data lives only in the two Member DCs. + +## Deployment topologies (2 DCs vs 3 DCs) + +The DR feature needs two things, in different quantities: + +- **Redis data** lives in the **Member** data centers (one active, one warm standby). +- **The failover decision** is made by the `dr-controlplane` etcd **quorum**. A quorum makes + progress only while a **majority of its three voting sites** is reachable. For single-fault + tolerance *and* split-brain safety, those three votes should sit in **three independent + failure domains**. The third domain is a tiny vote-only **Arbiter** (`role: Arbiter`) that + holds no Redis data. + +So "how many data centers" has two answers: how many hold **data** (two) and how many hold a +**quorum vote** (always three for automatic, split-brain-free failover). + +### A. Two Member DCs + an Arbiter DC (recommended) + +Three sites; two hold Redis data, the third is a vote-only Arbiter DC (`role: Arbiter`, no +Redis): + +```yaml +failoverPolicy: + mode: TwoDC +distributionRules: +- { clusterName: dc-east, role: Member, replicaIndices: [0, 1, 2] } +- { clusterName: dc-west, role: Member, replicaIndices: [3, 4, 5] } +- { clusterName: dc-arbiter, role: Arbiter } # etcd vote only, no Redis +``` + +Any single site can be lost: + +- **Lose a Member DC** then the surviving Member plus the Arbiter form a 2/3 majority, so the + survivor acquires the Lease and is promoted automatically; the lost DC, if alive but + partitioned, self-fences read-only. +- **Lose the Arbiter** then the two Members are still a 2/3 majority, so writes continue + uninterrupted. + +Because the Arbiter runs no Redis, it is small and cheap. **Run it in a third region or +cloud.** This is the lowest-cost way to get correct, automatic failover, and it is the +recommended topology whenever a third location is available. + +### B. Two sites only (reduced resiliency) + +If you genuinely have only two locations, you still need a third quorum vote, so you **place it +inside one of the two DCs** (run the third `dr-controlplane` etcd member there). There is no +separate Arbiter site, so that DC now holds **two of the three votes**: + +- **Lose the other DC** (the one with one vote) then the two-vote DC keeps the majority, so + failover and continuity work automatically. +- **Lose the two-vote DC** then the survivor holds only one of three votes, cannot form a + quorum, and therefore cannot safely become writable on its own. **Automatic failover does + not happen**; recovery is a manual, operator-confirmed step, and you must be certain the + failed DC is truly down to avoid split-brain. + +This protects against losing one specific DC, not both symmetrically. Prefer adding a cheap +third Arbiter site (topology A) whenever possible. + +### At a glance + +| Topology | Sites | Data DCs | Tolerates | Automatic failover | +| --- | --- | --- | --- | --- | +| Two Member + Arbiter (`TwoDC`) | 3 | 2 | any 1 site | yes | +| Two sites, co-located quorum | 2 | 2 | only the one-vote DC | only when the one-vote DC is lost | + +## Prerequisites + +- A working **distributed Redis** setup: Open Cluster Management (OCM) hub and spoke clusters, + KubeSlice connecting the spokes (it exports each DC's Redis endpoint cross-cluster for the + `REPLICAOF` link), and a storage class on each spoke. +- The `dr-controlplane` service and its three-site etcd quorum installed across the data + centers, with a `dr-controlplane` agent running in each spoke (DC). +- The KubeDB Redis operator started with the DC-DR flags: + + ``` + --dc-dr-enabled + --dc-dr-coord-kubeconfig= + --dc-dr-local-dc= + ``` + +- One consistent **DC name** per data center, used everywhere: the OCM spoke cluster name, the + agent `--dc-name`, the Lease `holderIdentity`, the marker `activeDC`, the pod label + `open-cluster-management.io/cluster-name`, and the `PlacementPolicy` + `distributionRule.clusterName`. Keep them identical. + +## Deploy a DC-DR Redis + +A DC-DR Redis is a distributed `Redis` whose `PlacementPolicy` carries a `failoverPolicy` and +per-DC roles. The user creates and edits a **single** `Redis` object and gets one `AppBinding` +and one connection endpoint; the operator expands it into the per-DC Redis deployments and +wires the `REPLICAOF` link between them. + +### 1. PlacementPolicy + +Assign the global pod ordinals to data centers and tag each DC with its role. Here two Member +DCs (`dc-east`, `dc-west`) each get three Redis pods, and `dc-arbiter` is the tie-breaking +vote: + +```yaml +apiVersion: apps.k8s.appscode.com/v1 +kind: PlacementPolicy +metadata: + name: redis-dcdr +spec: + clusterSpreadConstraint: + slice: + projectNamespace: kubeslice-demo + sliceName: demo-slice + failoverPolicy: + trigger: + scope: Global + mode: TwoDC + distributionRules: + - clusterName: dc-east + role: Member + replicaIndices: [0, 1, 2] + - clusterName: dc-west + role: Member + replicaIndices: [3, 4, 5] + - clusterName: dc-arbiter + role: Arbiter +``` + +- A data-bearing **Member** rule carries `replicaIndices`; the **Arbiter** DC (vote only, no + Redis) carries none. +- `failoverPolicy.trigger.scope: Global` makes this one cluster-wide failover scope. Use + `Group` with a group name to put a database in its own scope. + +### 2. Redis + +Reference the `PlacementPolicy` and opt the Redis into DC-DR expansion. This example uses +Sentinel mode: + +```yaml +apiVersion: kubedb.com/v1 +kind: Redis +metadata: + name: redis-dcdr + namespace: demo + annotations: + # Opt this distributed Redis into per-DC DC-DR expansion. + dr.kubedb.com/enabled: "true" +spec: + version: "7.4.0" + replicas: 3 + mode: Sentinel + sentinelRef: + name: sen-dcdr + namespace: demo + storageType: Durable + podTemplate: + spec: + podPlacementPolicy: + name: redis-dcdr + storage: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 1Gi + deletionPolicy: WipeOut +``` + +The operator then creates, per data-bearing DC: + +- a self-contained Redis (its own master, replicas, and Sentinel quorum) backed by a `PetSet` + named `-` (for example `redis-dcdr-dc-east`) with a DC-local governing `Service` + exported over KubeSlice, so peer discovery stays intra-DC; +- the cross-DC `REPLICAOF` link from each standby DC's master to the active DC's master (its + role-pinned primary `Service`, exported cross-DC over KubeSlice), with the active DC's master + `REPLICAOF NO ONE`. + +The Arbiter DC (`role: Arbiter`) runs no Redis pods. + +## Observe the DC-DR state + +The single `Redis` object's `status.disasterRecovery` carries the whole cross-DC view: + +```bash +$ kubectl get redis -n demo redis-dcdr -o jsonpath='{.status.disasterRecovery}' | jq +``` + +```json +{ + "activeDC": "dc-east", + "phase": "Steady", + "dataCenters": [ + { + "clusterName": "dc-east", "role": "Member", + "master": "redis-dcdr-dc-east-0", + "writable": true, "healthy": true + }, + { + "clusterName": "dc-west", "role": "Member", + "master": "redis-dcdr-dc-west-0", + "writable": false, "linkStatus": "up", + "lagBytes": 4096, "healthy": true + } + ] +} +``` + +- `activeDC` is the DC that currently holds the Lease and runs the one writable master. +- `phase` is `Steady`, `FailingOver`, `FailingBack`, or `Degraded`. +- Each `dataCenters` entry reports that DC's master pod, whether it is writable, its cross-DC + `linkStatus` (`master_link_status` from `INFO replication`, empty on the active DC), its + `lagBytes` (the active master's `master_repl_offset` minus this DC master's replicated + offset), and whether it is healthy. The in-DC coordinator computes these from + `INFO replication` and publishes them as pod annotations; the hub never opens a cross-DC + Redis connection. + +## Unplanned failover + +When the active DC is lost, its agents stop renewing the primary-DC Lease. After the Lease +duration the surviving Member DC's agent acquires it; that DC becomes `activeDC`. The hub +observes the change and records it in status; the survivor's fence then clears, the operator +runs `REPLICAOF NO ONE` on the survivor's master to promote it, relabels it `primary`, and +re-points any other standby DC at the new active master. The old DC, if partially alive, has +already self-fenced (its master stays `standby`, read-only). The primary `Service` and +`AppBinding` then resolve to the new writable DC. + +You do not trigger this; it is automatic. `status.disasterRecovery.phase` moves to +`FailingOver` during the transition and back to `Steady` once the survivor is primary. Because +Redis replication is asynchronous (acked by offset), the RPO is bounded by the survivor's +cross-DC offset lag at the moment the active DC died (the un-shipped replication tail). + +## Planned switchover (near-zero-RPO) + +To move the active DC on purpose (maintenance, rebalancing) without losing writes, annotate the +Redis with the target DC: + +```bash +$ kubectl annotate redis -n demo redis-dcdr dr.kubedb.com/switchover-to=dc-west +``` + +The switchover is coordinated for near-zero RPO: + +1. The target must be a known, healthy DC within the lag budget + (`dr.kubedb.com/switchover-max-lag-bytes`, default 16Mi). +2. The hub quiesces writes on the active DC's master (it is held read-only through the Lease + quiesce marker), so its `master_repl_offset` stops advancing. +3. The hub waits until the target master's replicated offset reaches the quiesced active + master's `master_repl_offset` (within a small near-zero-RPO window). +4. The hub hands off the Lease to the target DC. The target is promoted (`REPLICAOF NO ONE`), + the old primary becomes a `REPLICAOF` replica of the new active, and the annotation is + cleared automatically. + +Because the active master is quiesced and the target reaches offset equality before the handoff, +a planned switchover loses no acknowledged writes. + +## Failback + +A returned old-active DC becomes a `REPLICAOF` replica of the new active. Redis reconciles a +diverged replica by partial resync from the replication backlog when possible, otherwise a full +RDB resync that discards the replica's local data and re-seeds it, so there is no rewind to +implement; the diverged tail is simply dropped by the full resync. After catch-up, a +coordinated planned switchover returns the active DC. + +## Scale a data center + +Each DC has its own intra-DC Redis, so a single `spec.replicas` cannot describe a scale. Scale a +specific DC with a `RedisOpsRequest` that lists per-DC targets (each entry sets that DC's local +node count; DCs not listed are unchanged), then the operator updates the `PlacementPolicy` so +the declarative topology matches. No other DC and no cross-DC writability is touched. + +## Cleanup + +```bash +$ kubectl delete redis -n demo redis-dcdr +$ kubectl delete placementpolicy redis-dcdr +``` + +Deleting the `Redis` removes the per-DC `PetSet`s, governing `Service`s, and the cluster-scoped +per-DC `PlacementPolicies` the operator generated. The user-provided base `PlacementPolicy` is +left for you to delete. diff --git a/docs/guides/redis/dr/runbook/index.md b/docs/guides/redis/dr/runbook/index.md new file mode 100644 index 0000000000..bf2bc744e1 --- /dev/null +++ b/docs/guides/redis/dr/runbook/index.md @@ -0,0 +1,256 @@ +--- +title: DC-DR Runbook +menu: + docs_{{ .version }}: + identifier: rd-dr-runbook + name: Runbook + parent: rd-dr + weight: 30 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +# Redis DC-DR Runbook + +Scenario-by-scenario procedures for operating a Redis (or Valkey) cluster in cross data center +disaster recovery (DC-DR) mode. Each scenario lists the **symptoms**, what KubeDB does +**automatically**, how to **verify**, and the **action** to take. + +Read the [DC-DR Overview](/docs/guides/redis/dr/overview/index.md) for the concepts and +commands referenced here. Throughout, `` is the coordination control plane kubeconfig, +`redis-dcdr`/`demo` are the example database and namespace. + +## Quick reference + +```bash +# Active DC, phase, and per-DC view: +kubectl get redis -n demo redis-dcdr -o jsonpath='{.status.disasterRecovery}' | jq + +# Lease holder (the source of truth for "who is active"): +kubectl --kubeconfig -n dc-failover get lease primary-dc -o jsonpath='{.spec.holderIdentity}' + +# Per-DC masters, roles, and DCs: +kubectl get pods -n demo -l app.kubernetes.io/instance=redis-dcdr -L kubedb.com/role,open-cluster-management.io/cluster-name + +# A spoke's marker (what its coordinators read): +kubectl -n dc-failover get configmap primary-dc -o jsonpath='{.data}' +``` + +Golden rules: + +- **The Lease decides who is writable.** Never run `REPLICAOF NO ONE` or a manual + `SENTINEL FAILOVER` across DCs by hand. +- **The fence fails closed.** A DC that cannot confirm it holds the Lease keeps its master + labeled `standby` and read-only by design; that is correct, not a bug. +- **Exactly one DC is `writable: true`** in `status.disasterRecovery` at any instant. + +--- + +## 1. Active DC lost (zone/cluster failure) + +**Symptoms:** the active DC's pods are gone/unreachable; writes fail briefly. + +**Automatic:** the lost DC's agents stop renewing the Lease. After the Lease duration (~45s) +the surviving Member DC's agent acquires it and becomes `activeDC`. The hub records the change; +the survivor's fence clears, the operator runs `REPLICAOF NO ONE` on the survivor's master to +promote it, relabels it `primary`, and re-points any other standby DC at it. The old DC, if +partially alive, has already self-fenced (its master stays `standby`, read-only). The primary +`Service` and `AppBinding` follow to the new DC. `phase` moves `FailingOver` to `Steady`. + +**Verify:** + +```bash +kubectl get redis -n demo redis-dcdr -o jsonpath='{.status.disasterRecovery.activeDC}' # the survivor +kubectl get redis -n demo redis-dcdr -o jsonpath='{.status.disasterRecovery.phase}' # Steady +``` + +**Action:** none required for availability. Note the RPO: the replication tail not yet shipped +when the DC died (the offset gap) is lost, because Redis cross-DC replication is asynchronous. +When the failed DC returns, see scenario 9 (re-add a DC). + +--- + +## 2. Network partition between data centers + +**Symptoms:** DCs are up but cannot reach each other or the coordination plane. + +**Automatic:** the side that loses the coordination plane stops getting Lease updates; its +marker `renewTime` freezes and, after the 30s fence TTL, its `rd-coordinator` keeps its master +labeled `standby` (read-only), **before** the Lease duration lets the other side acquire it +(this is the timing invariant). The side that keeps the etcd majority holds/acquires the Lease +and stays (or becomes) writable. There is no split-brain. + +**Verify there is exactly one writable DC:** + +```bash +kubectl get redis -n demo redis-dcdr -o jsonpath='{range .status.disasterRecovery.dataCenters[*]}{.clusterName}={.writable} {end}' +``` + +**Action:** heal the network. The fenced side re-points its master `REPLICAOF` the active DC and +resumes its stream automatically (or full-resyncs if it diverged). If both sides show +`writable: false`, see scenario 6 (coordination plane down). + +--- + +## 3. Planned switchover (maintenance on the active DC) + +**Action:** + +```bash +kubectl annotate redis -n demo redis-dcdr dr.kubedb.com/switchover-to=dc-west +``` + +**Automatic:** the hub gates on the target's health and offset lag, quiesces writes on the +active master (holds it read-only through the Lease quiesce marker), waits until the target +master's replicated offset reaches the active master's `master_repl_offset`, then promotes the +target (`REPLICAOF NO ONE`), makes the old primary a `REPLICAOF` replica of the new active, and +hands off the Lease. No acknowledged writes are lost. The annotation is cleared on completion. + +**Verify:** + +```bash +kubectl get redis -n demo redis-dcdr -o jsonpath='{.status.disasterRecovery.activeDC}' # dc-west +kubectl get redis -n demo redis-dcdr -o jsonpath='{.status.disasterRecovery.phase}' # Steady +``` + +**If it does not complete:** see scenario 7 (switchover stuck). + +--- + +## 4. Planned failback to the original DC + +After the original DC is healthy and caught up: + +```bash +kubectl annotate redis -n demo redis-dcdr dr.kubedb.com/switchover-to=dc-east +``` + +Same near-zero-RPO flow as scenario 3. A DC that previously lost the Lease rejoins as a +`REPLICAOF` replica and catches up (partial resync, or a full RDB resync that drops the diverged +tail, see scenario 9) before it is eligible, so failback is safe even after an unplanned +failover. + +--- + +## 5. A standby DC is lost + +**Symptoms:** the non-active DC's pods are gone; that DC shows `healthy: false`. + +**Impact:** none on writes, the active DC is unaffected. You lose your only DR copy and the +standby read capacity until it returns. + +**Automatic:** nothing to fail over. The active master's replication backlog is bounded, so a +long-gone standby does not grow the active master's memory without limit. + +**Action:** bring the DC back. When it returns it re-points `REPLICAOF` the active master and +resumes streaming (partial resync if the backlog still covers its offset, otherwise a full RDB +resync). Watch `linkStatus` return to `up` and `lagBytes` fall in +`status.disasterRecovery`. + +--- + +## 6. Coordination plane (etcd quorum) down + +**Symptoms:** no DC can renew or acquire the Lease; every DC's marker goes stale. + +**Automatic:** every DC fails closed. After the 30s fence TTL, all masters are held `standby` +(read-only). Writes stop everywhere. This is the safe state: with no quorum, no DC can prove it +is the single writable DC. + +**Verify:** + +```bash +kubectl --kubeconfig -n dc-failover get lease primary-dc # stale or missing renewTime +``` + +**Action:** restore the `dr-controlplane` etcd quorum (a majority of its three sites). As soon +as a DC's marker refreshes and names it active, its fence lifts and it resumes as the writable +primary. Do **not** force a master writable by hand while the quorum is down. + +--- + +## 7. Switchover stuck (`phase: FailingOver` persists) + +**Symptoms:** after annotating `dr.kubedb.com/switchover-to`, the active DC does not move. + +**Likely causes and checks:** + +- **Target lag over budget.** The hub will not hand off until the target is within + `dr.kubedb.com/switchover-max-lag-bytes` (default 16Mi). Check the target's `lagBytes` and + `linkStatus`: + + ```bash + kubectl get redis -n demo redis-dcdr -o jsonpath='{range .status.disasterRecovery.dataCenters[*]}{.clusterName}: link={.linkStatus} lag={.lagBytes}{"\n"}{end}' + ``` + + Wait for the standby to catch up, or raise the budget annotation if appropriate. +- **Target not healthy.** The target DC must show `healthy: true`. +- **Write pressure.** A very high write rate can keep the target from reaching offset equality; + the quiesce freezes the active offset, so it should converge. If it does not, reduce load and + retry. + +**Action:** to abort, remove the annotation: + +```bash +kubectl annotate redis -n demo redis-dcdr dr.kubedb.com/switchover-to- +``` + +The active DC stays where it is and the quiesce is cleared. + +--- + +## 8. Marker stale on a healthy DC + +**Symptoms:** a DC's master is unexpectedly held `standby`; its marker `renewTime` is old. + +**Automatic:** this is the fence working. The `rd-coordinator` refuses to promote a master whose +`primary-dc` marker is missing, unparseable, stale (older than 30s), or names another DC. + +**Verify:** + +```bash +kubectl -n dc-failover get configmap primary-dc -o jsonpath='{.data}' # activeDC + renewTime +``` + +**Action:** find why the DC's `dr-controlplane` agent is not refreshing the marker (agent down, +RBAC on the `dc-failover` ConfigMap, or the DC genuinely lost the Lease). Fix the agent or the +connectivity; the marker refreshes and the fence lifts on its own. + +--- + +## 9. Re-add a data center after it returns + +**Symptoms:** a previously lost DC is back but was the old active DC, so its data may have +diverged from the new active. + +**Automatic:** the operator makes the returned DC a `REPLICAOF` replica of the new active. Redis +reconciles it by partial resync from the backlog if possible, otherwise a full RDB resync that +**drops the diverged tail** and re-seeds from the active master. There is no manual rewind. + +**Verify it caught up:** + +```bash +kubectl get redis -n demo redis-dcdr -o jsonpath='{range .status.disasterRecovery.dataCenters[*]}{.clusterName}: link={.linkStatus} lag={.lagBytes}{"\n"}{end}' +``` + +**Action:** once `linkStatus: up` and `lagBytes` is small, the DC is a healthy standby again. To +return the active role to it, run a planned failback (scenario 4). + +--- + +## 10. Verifying the split-brain guarantee + +At any instant there must be exactly one writable DC. To assert it: + +```bash +# From status (what the hub sees): +kubectl get redis -n demo redis-dcdr -o jsonpath='{range .status.disasterRecovery.dataCenters[*]}{.clusterName}={.writable} {end}' + +# From the pods (what the role labels say): exactly one primary across all DCs. +kubectl get pods -n demo -l app.kubernetes.io/instance=redis-dcdr,kubedb.com/role=primary -L open-cluster-management.io/cluster-name +``` + +If ever two DCs report `writable: true` or two pods across DCs carry `kubedb.com/role: primary`, +capture the Lease (`kubectl --kubeconfig -n dc-failover get lease primary-dc -o yaml`) +and both spokes' `primary-dc` markers and open an issue; the fence and the etcd majority are +designed to make this impossible.