Shared multi-cloud SRDP deployment layer

[yannick-vinkesteijn]

Feature description

A shared, provider agnostic layer that installs the SRDP app and is reused across clouds, so each cloud only supplies its own infra values. Split out from #34.

Use case

The Scaleway blueprint bundled the app install under deploy/scaleway/deploy/srdp/. That ties the app deployment to one cloud and hardcodes infra values (S3 URL, Secret Manager region). We want to write the app install once and reuse it across providers instead of copying it per cloud.

Removed overlay for reference (pinned commit, before the strip): https://github.com/srdp-hub/srdp/tree/6571735939e43fbf8d44b97a9fb4d25283f6234f/deploy/scaleway/deploy/srdp

Proposed solution

Two nested layers:

deploy/
  docker/                Compose stack (local), already here
  kubernetes/
    srdp-chart/          the chart (the contract), already here
    clusters/<env>/      Flux delivery overlay: values + HelmRelease, cloud agnostic (new)
  scaleway/              Scaleway infra (Terraform), its Flux points at deploy/kubernetes/clusters/<env>
  azure/ (future)        different infra, same deploy/kubernetes/clusters

Convention: deploy/<runtime> is self-contained (like deploy/docker for Compose). The k8s app deployment lives under deploy/kubernetes next to the chart, not in a separate top-level folder. Cloud folders only provision infra and point Flux at it.

Contract that keeps the app layer cloud agnostic: each provider's Terraform writes its outputs (DB host, S3 URL, project, region, secret refs) into a ConfigMap. Flux injects them into the shared values with postBuild.substituteFrom. No cloud specific values live in the app layer.

Chart as contract: deployment values only set infra knobs (hosts, domains, tiers, buckets). Do not restate core config (db names, ports, ETL env vars) that already lives in the chart. Relates to the DuckLake env contract backlog item.

Tasks:

• Create deploy/kubernetes/clusters/<env> shared overlay (base + per env)
• Define the Terraform outputs to ConfigMap to substituteFrom handoff
• Re-add the SRDP install with the review fixes: Dagster ["-m", "etl.definitions"], LAKEHOUSE_S3_URL via substitution not hardcoded, immutable image tags for prod
• Point Scaleway's Flux at deploy/kubernetes/clusters/<env>
• Document the layering as the blueprint convention in deploy/README.md

Alternatives considered

Keep the app install per provider: duplicated config that drifts. Restructure it inside #34: blows up that PR's scope and forces the handoff contract under merge pressure.

Component: Deploy: Kubernetes, Deploy: Other

[yannick-vinkesteijn]

Carrying over the app wiring that was removed from PR #34 with the srdp overlay (design notes are in that PR's history). The shared app layer needs to cover:

• Existing secrets: consume srdp-postgres, srdp-zitadel, srdp-oauth2, srdp-lakehouse as Kubernetes secrets synced from the cloud secret manager by External Secrets. The chart may need existingSecret support for the Zitadel masterkey and DB password (today the example values inline them).
• Dagster code location: the old values pointed at module definitions, but fix: correct Dagster module name in values-local.yaml #29 moved it to etl.definitions. Use the current path, ideally sourced from the platform config so it can't drift again.
• Internal Traefik load balancer: confirm the Scaleway annotations (scw-loadbalancer-private, scw-loadbalancer-pn-ids) give a load balancer that is only reachable on the private network (no public IP), and that Zitadel and oauth2-proxy still resolve and redirect correctly when accessed over the mesh.
• DuckLake env contract: align the srdp-etl env vars (DUCKLAKE_PG_*, LAKEHOUSE_S3_URL, AWS_*) with src/srdp/io/ducklake.py and src/srdp/io/storage.py.
• Terraform outputs to values: the old values hand-copied Terraform outputs into placeholders. Better: Terraform seeds an infra-outputs ConfigMap that the HelmRelease reads via valuesFrom, so nothing is copied by hand.

[yannick-vinkesteijn]

Heads up: the proposed layout in this issue predates PR #34. The shared Flux tree it describes (deploy/kubernetes/clusters/<env>) now exists as deploy/gitops/ with platform/base (cloud-agnostic) and clusters/<cloud>/<env> leaves, and each cloud's Flux already points at its leaf. See deploy/gitops/README.md for the design rules.

So for this issue that means:

• The app layer lands as a sibling of platform/ inside deploy/gitops/ (e.g. deploy/gitops/srdp/base plus per-cloud/per-env values in the cluster leaves), not under deploy/kubernetes/.
• The task "Point Scaleway's Flux at deploy/kubernetes/clusters/<env>" is obsolete, Flux already reconciles the gitops leaves.
• deploy/kubernetes itself is planned to become deploy/charts in deploy/ cleanup: OpenTofu and functional layout #44.

The Terraform-outputs-to-ConfigMap handoff and the chart-as-contract points stand unchanged.