Proposal: srdp.toml, a single platform config

[yannick-vinkesteijn]

Problem

Platform configuration is currently scattered over services/deployment specific env and config files, user needs to manually copy files and update multiple files. This increases the risk of drift and configuration issues.

Proposal

A single repo-level config file (working name srdp.toml) as the platform contract:

• Holds topology and identity: org prefix, cloud, environment set, services on/off, code locations, CI policy.
• Schema is a pydantic model, same pattern as the existing DuckLakeSettings: validation, error messages, and JSON Schema for free.
• Never holds secret values. Secrets are stored in .env, secrets.toml and/or secret managers.
• Ownership rule: We need to define clear boundary between optional services and components available in core and outside what srdp provides (plugins) and make sure both can be accommodated in the config.
• File first, CLI second: hand-editable; a future srdp CLI (srdp init, srdp env add) reads and writes the same file and keeps no other state. Enables interactive setup/deployment later, including scaffolding env dirs instead of shipping three examples.
• The config file is a blueprint of the environment and should be sufficient to setup and deploy the srdp instance from scratch.

Consumers: Justfile, CI workflow (dynamic matrix + gate), the app deployment layer (#38), the future CLI.

Open questions

• TOML vs YAML (TOML proposed: strict types, stdlib parsing, comment-preserving edits for the CLI; the schema is format-agnostic). TOML is also used in libraries like DLT and streamlit. Both can be natively parsed to a dictionary and validated by pydantic

Relates to #38, PR #34 review discussion.

[AE-EVKA]

Strongly in favour, and I may be able to help with one of the open questions.

The issue lists "ownership boundaries between optional services, core components, and external
plugins" as still undecided. I've been building that third category for the past few weeks —
datavloot deployed as a client project on the Compose stack — so I can describe what an external
plugin actually needs to declare, from a real case rather than a hypothetical one.

Today I express all of it as a Compose overlay layered on deploy/docker/docker-compose.yml, pinned
to an exact commit. It works, but it means my integration surface is your file layout plus Compose
merge semantics, which has a failure mode I'd like to design away: rename a service or a network
upstream and my overlay still produces valid YAML that merges into nothing. First symptom is a 404
from Traefik, three layers from the cause. srdp.toml replaces that with a schema — testable, and
breakable with a real error message.

What my overlay currently has to say, which is roughly the shape of what a plugin would declare:

• an app — a service to put behind the SSO gate, with a hostname and a port (this is the one
  with no mechanism today; I hand-write Traefik labels against your middleware names)
• a code location — image, module, Dagster version (see Improve configuration and onboarding re: Dagster code locations #23)
• storage — DuckLake metadata schema, data path, and local-vs-S3
• roles — which of ADR-0005's reader/writer/admin gate what
• extra OIDC scopes — I need urn:zitadel:iam:org:projects:roles in the access token, and today
  the only way to add it is to copy your whole oauth2-proxy command

[yannick-vinkesteijn]

Proposal for the schema, from working on #64:

• One pydantic root model for the whole file, with extra="forbid" at every level, so a typo fails at load time.
• Consumers such as the setup service read their part from that model.
• Services are typed entries, validated as a discriminated union on type.
• The CLI edits the file with tomlkit, so comments survive.

Versioning is tracked in #68.