Skip to content

Add DC-DR types to Kafka - #1798

Open
tamalsaha wants to merge 1 commit into
masterfrom
kafka-dc-dr
Open

tamalsaha wants to merge 1 commit into
masterfrom
kafka-dc-dr

Conversation

@tamalsaha

Copy link
Copy Markdown
Member

Adds the distributed cross data center (DC-DR) API surface to Kafka, mirroring the Postgres and MongoDB DC-DR types.

What

  • spec.distributed and spec.podPlacementPolicy on KafkaSpec, so one Kafka CR expands into one self-contained Kafka cluster per Member data center.
  • status.disasterRecovery (KafkaDisasterRecoveryStatus, KafkaDCStatus, KafkaDRPhase) reporting the active DC, per DC broker health, cross-DC MirrorMaker 2 mirror lag in milliseconds (mirrorLagMillis), and the DR phase (Steady, FailingOver, FailingBack, Degraded).
  • Per data center dataCenters on KafkaHorizontalScaling ops, so each DC's local Kafka cluster is scaled independently.

Notes

  • Kafka DC-DR is active/passive: exactly one data center is the write cluster, chosen by the dr-controlplane primary-DC Lease, and MirrorMaker 2 asynchronously mirrors it to the standby. Lag is expressed in milliseconds (MM2 replication latency), not bytes.
  • Hand-spliced deepcopy and the spec and ops CRD fields. Full CRD and openapi regeneration is left to make gen in CI, matching the sibling Postgres and MongoDB DC-DR PRs.
  • go build ./apis/kubedb/v1/... ./apis/ops/v1alpha1/... is green and the changed files are gofmt clean.

Adds the distributed cross data center (DC-DR) API surface to Kafka, mirroring
the Postgres and MongoDB DC-DR types:

- spec.distributed and spec.podPlacementPolicy on KafkaSpec, so one Kafka CR
  expands into one self-contained Kafka cluster per Member data center.
- status.disasterRecovery (KafkaDisasterRecoveryStatus, KafkaDCStatus,
  KafkaDRPhase) reporting the active DC, per DC broker health, cross-DC
  MirrorMaker 2 mirror lag in milliseconds, and the DR phase.
- Per data center dataCenters on KafkaHorizontalScaling ops, so each DC's
  local Kafka cluster is scaled independently.

Hand-spliced deepcopy and the spec and ops CRD fields. Full CRD and openapi
regeneration is left to make gen in CI.

Signed-off-by: Tamal Saha <tamal@appscode.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant