Skip to content

engine-providers.md

Latest commit

 

History

History
311 lines (265 loc) · 22.1 KB

engine-providers.md

File metadata and controls

311 lines (265 loc) · 22.1 KB
Raw

Select an independently operated engine

A product can wait for an externally owned SQL, Document or Graph engine and its published application credentials. The controller observes readiness; the database operator creates, upgrades, backs up and deletes the source. Product workloads consume the connection Secret directly.

Enable both provisionedSources.enabled=true and engineProviders.enabled=true in the Helm chart. The manager settings are PROVISIONED_SOURCES_ENABLED=true and ENGINE_PROVIDERS_ENABLED=true. The OpenFeature flags provisioned-sources and engine-providers default off. A declared engine stays unready while either gate is off, and disabled gates perform no source or Secret reads. Apply the release's CRD before upgrading an existing chart installation, as described in the README.

Supported selection

spec.source.engine uses the engine-provider/v1 contract. Admission binds each supported selection to one adapter and resource API. The runtime resolver repeats that check before any reads.

Selection Adapter Referenced resource Connection publication
Engine omitted crossplane/v1 Namespaced custom resource Matching writeConnectionSecretToRef, owned by that resource
sql / native cnpg/v1 postgresql.cnpg.io/v1 Cluster Operator-generated <cluster>-app Secret, owned by the current Cluster UID
document / native percona-mongodb/v1 psmdb.percona.com/v1 PerconaServerMongoDB Explicit custom-user password Secret, published with the current source UID
graph / native arangodb/v1 database.arangodb.com/v1 ArangoDeployment Independently published read-only application password, with v1 metadata and current source UID
document / cnpg-hybrid cnpg-hybrid/v1 postgresql.cnpg.io/v1 Cluster Dedicated JSONB reader publication, current source UID and generation
graph / cnpg-hybrid cnpg-hybrid/v1 postgresql.cnpg.io/v1 Cluster Dedicated AGE reader publication, current source UID and generation

SQL uses the native adapter; a SQL hybrid selection is unsupported. Unknown versions, contradictory adapters and other resource APIs are rejected at admission. SQL requires its generated application Secret name; Document and Graph publication receive additional runtime checks against the externally owned source. Existing untyped Crossplane products retain their contract.

The SQL example refers to a Cluster named warehouse in products. Install and configure CloudNativePG independently, including storage, database ownership, network access, backups and retention. Deploy the product's query workload and public contract independently; declaring output URLs does not create them.

CloudNativePG observation contract

The cnpg/v1 adapter requires:

  • exactly one Ready=True condition;
  • positive spec.instances, with matching status.instances and status.readyInstances;
  • matching, nonempty status.currentPrimary and status.targetPrimary;
  • current explicit observed generations; an omitted generation follows CNPG's condition contract;
  • a non-deleting generated application Secret whose owner reference matches the current Cluster's UID, name, API group and kind.

The adapter accepts the operator-generated application Secret only. A supplied spec.bootstrap.initdb.secret, spec.bootstrap.recovery.secret or spec.bootstrap.pg_basebackup.secret is outside this publication contract and reports ConnectionPublicationUnsupported. Application workloads use the generated credentials directly; the controller never requests their values. See CloudNativePG's application connection guide and Cluster API reference.

SourceReady refreshes independently of other dependencies. Missing sources, publication ownership mismatches, lost permissions, unready replicas and deletion produce stable reasons without copying provider messages into product status. Aggregate readiness also requires the product's other declared dependencies. Typed observation uses a fixed resource mapping, fresh exact-name reads, a five-second deadline and 30-second polling; unchanged observations do not rewrite status. After a selected provider completes, cancellation or deadline expiry makes the observation unavailable, even if the final API read returned a healthy object. This completion check also applies to the legacy Crossplane adapter.

Engine and legacy source readers negotiate only single-object partial metadata for application Secrets. Servers that cannot provide that representation fail observation with SourceUnavailable; the client never negotiates a full-Secret fallback. Kubernetes still authorizes the complete Secret GET, so exact-name RBAC remains required. Normal source-object reads retain their full operator status representation.

CloudNativePG may omit observed generations. In that case the observer cannot establish that status reflects the latest spec. Control-plane readiness and Secret ownership do not prove Secret contents, database authentication, query availability, backups or extension support. The registry publishes neither source references nor credentials.

PostgreSQL hybrid observation contract

The hybrid products refer to an independently operated CloudNativePG Cluster. Document uses JSONB; Graph uses the owned PostgreSQL / AGE image. Both require the native Cluster readiness checks, an exact supported immutable image in spec.imageName, and the same currently running image reported in status.image. Catalog indirection cannot establish this profile. Document also supports the upstream minimal PostgreSQL 17.11 Trixie index. Graph requires the owned AGE 1.7.0 profile and age in the operator's spec.postgresql.shared_preload_libraries array. A free-form parameter or invented status field does not satisfy the declared preload profile. The controller neither initializes the extension nor verifies its installation by connecting.

The supported immutable profiles are:

  • Core JSONB: ghcr.io/cloudnative-pg/postgresql:17.11-minimal-trixie@sha256:d78e771decf39071aa8bfb96684e8b7e6e5f3c6e00a945404249756db2c6c712.
  • JSONB or AGE: ghcr.io/devantler-tech/data-product-controller-postgresql-age:17.11-age1.7.0-dpc1.16.0@sha256:0b6e2d75d5551586570979d767a28b255953c2ee86820409ad9fe37d91ce3fa8.

Both the version tag and digest are required: CloudNativePG uses the tag to detect upgrades, while the digest fixes the image bytes. Digest-only references, other tags, digests and repositories are unsupported, including newer owned images until their profile is validated. Real PostgreSQL acceptance uses CloudNativePG 1.30.1; source observation does not verify the installed operator binary. The owned image's v1.16.0 source is 04d6ec7b517b59e6d2cffcab484f8a42a711c1ce, signed by the immutable publisher 86f0f95e5ac93ec914f5717f561af878b7d09bf1 under the verification procedure in the image guide.

Choose the hybrid provider for PostgreSQL storage and operating conventions, with these explicit capability differences:

Model Hybrid query capability Native provider difference
Document JSONB values in PostgreSQL tables, with a publisher-selected schema, table and column It does not provide MongoDB wire-protocol compatibility, MongoDB operators or native document command semantics
Graph AGE Cypher over graph tables in the selected PostgreSQL database It does not expose ArangoDB AQL or its native graph/document APIs
Lifecycle Both models share the independently operated CNPG source, storage and failure domain Dedicated native engines have their own operators, configuration and source lifecycle

Query contracts describe each independent application interface; selecting a model does not make native and hybrid query languages interchangeable. AGE requires supported preload configuration, extension initialization in the selected database and a planned restart for existing sources. The owned image guide describes those steps. Real acceptance verifies both query languages and reader roles; metadata-only observation cannot establish them.

Use dedicated reader Secrets. Bootstrap-owner, superuser, replication, server, CA and client credential names are rejected before any reads. Configured superuser, bootstrap application and certificate references are also excluded after reading the Cluster and before reading publication metadata. An independent publisher verifies application queries and effective read-only grants, then adds metadata and the current Cluster owner reference:

metadata:
  annotations:
    data.devantler.tech/cnpg-hybrid-publication: v1
    data.devantler.tech/cnpg-hybrid-access: read-only
    data.devantler.tech/cnpg-hybrid-source-generation: "3"
    data.devantler.tech/cnpg-hybrid-database: catalog
    data.devantler.tech/cnpg-hybrid-user: document_reader
    data.devantler.tech/cnpg-hybrid-capability: jsonb/v1
    data.devantler.tech/cnpg-hybrid-schema: public
    data.devantler.tech/cnpg-hybrid-table: documents
    data.devantler.tech/cnpg-hybrid-column: payload

For Graph, use capability age/1.7.0 and data.devantler.tech/cnpg-hybrid-graph: lineage. Identifiers contain 1–63 lowercase ASCII letters, digits or underscores and start with a letter. Privileged PostgreSQL and CNPG users are unsupported. A changed Cluster generation requires a fresh publication after query verification; recreation also requires the new source UID. Rotation at the same publication name uses the application workload's projected password without a controller restart.

The hybrid observer Role grants only the named Cluster and dedicated publication GETs. Metadata declares publisher intent; it does not prove Secret contents, effective privileges, query health, graph existence, backups or PostgreSQL extension compatibility.

Grant narrow observation access

Apply the observer Role example in the product namespace. Adjust its controller ServiceAccount and namespace for your installation. It grants get only on warehouse and warehouse-app; the chart does not grant database or Secret access.

The manager requests PartialObjectMetadata for the Secret and bypasses caches. Kubernetes RBAC authorizes a whole Secret get, even when the request negotiates metadata only, so keep resource names and namespaces explicit. No list, watch or mutation grant is needed for external sources.

Percona MongoDB observation contract

The Document product selects the percona-mongodb/v1 adapter. Install Percona Operator for MongoDB 1.23.0 independently and use spec.crVersion: 1.23.0. That declaration does not verify the installed operator image. The initial supported profile is one managed, unpaused, unsharded replica set, with positive size and no arbiter, non-voting, hidden or external members. status.state must be ready, and status.size and status.ready must equal the requested size. An explicit status.observedGeneration must be current. Percona's published status contract may omit it; in that case the observer cannot establish that status reflects the latest spec.

The referenced password Secret must match exactly one explicit spec.users[].passwordSecretRef.name. The selected custom user must declare only built-in read roles on non-system databases. Its authentication database may be admin; $external authentication is unsupported. For example:

spec:
  crVersion: 1.23.0
  replsets:
    - name: rs0
      size: 3
  users:
    - name: catalog-reader
      db: admin
      passwordSecretRef:
        name: documents-reader
        key: password
      roles:
        - name: read
          db: catalog

This fragment shows the observation fields, not a complete database installation. An independent publisher must bind documents-reader to the current PerconaServerMongoDB through an owner reference with API psmdb.percona.com/v1, kind, name and UID. This is a controller publication requirement; Percona documentation does not promise that ownership automatically. Do not modify operator-managed system or connection-string Secrets to satisfy it. Admission rejects the default system Secret. Initial runtime validation also rejects it as SourceInvalid, before any reads. Configured operator credential roles under spec.secrets and spec.vault, internal system Secrets, operator-generated passwords, connection-string Secrets, system accounts, duplicate usernames, multiple users sharing the password Secret, custom roles and privileged roles report ConnectionPublicationUnsupported. Reserved publication names follow Percona's connection-Secret naming contract; a manual password binding cannot reuse another declared user's generated connection-Secret name. The supported operator's Secrets and Vault API sections define these credential roles; their references are checked before any application Secret metadata read.

Apply the Document observer Role, adjusting its ServiceAccount for your installation. It grants only the named source and password Secret GETs. The application workload consumes its credentials independently; the controller requests only Secret metadata. Declared read roles and publication ownership do not prove effective privileges, password validity, database access or query availability. See Percona's custom-user guide, status contract and system-user guidance.

ArangoDB Graph observation contract

The Graph product selects arangodb/v1. Its query URL and OpenAPI document describe a separately operated workload; DPC does not create that workload or send AQL queries. Install the ArangoDB operator independently, using the 1.4.5 API. The real-provider acceptance profile requires spec.mode: Single, explicit spec.single.count: 1, enabled authentication and the verified release index sha256:4bc086d5050ca7ea11c6d00a36d8b910c838bb54ad553f8c1b715769d3499bcf. The index is accepted with the 3.12.12 tag or without a tag, using arangodb or docker.io/library/arangodb. The original tag-only declaration arangodb:3.12.12 retains its previous non-enterprise binary-metadata requirement; it does not identify the immutable profile tested here. Other digests, registries, image repositories and contradictory tags are unsupported. Single mode provides no high availability.

The live typed specification checksum must match both status.acceptedSpecVersion and status.appliedVersion. DPC does not apply defaults before hashing: the operator hashes the raw spec and separately stores defaulted status.accepted-spec. Hashing preserves the pinned operator's Kubernetes 0.33 PVC metadata encoding, including a null empty creation timestamp; the controller's newer Kubernetes library otherwise omits that field. The accepted specification must also retain exactly the declared image, Single/count-one profile and a resolved authentication Secret; contradictory status does not establish readiness. Ready, SpecAccepted, UpToDate, BootstrapCompleted and upstream's misspelled BootstrapSucceded conditions must be True. Deployment phase must be Running, with exactly one Created and Ready Single member, a modern Pod name/UID, matching image declarations and reported desired/running image IDs, and ArangoDB 3.12.12 versions. The verified official immutable index reports an Enterprise binary marker; both current and member observations must match that actual binary profile. The marker does not establish license entitlement. Update, upgrade, Secret-change, pending update, member-restart and member-termination states withdraw readiness, even when the departing member still reports Ready. Missing, unknown, malformed or duplicate conditions cannot establish readiness. Condition hashes, transition timestamps, historical SpecPropagated and Pod-spec checksums are not freshness markers.

An independent publisher owns application setup and the password Secret. The pinned bootstrap validator accepts only root accounts. The publisher must create a dedicated non-administrator user, deny _system and database wildcard access, grant ro on the application database, deny collection wildcard access and grant ro on every named vertex/edge collection. Assign explicit none grants to all other application collections. ArangoDB's database ro grant otherwise supplies read access when no specific collection grant exists; a collection wildcard none does not override it. The publisher must install a specific denial before adding an unpublished collection, and keep system collections outside the published query contract. It publishes the password for consumption directly by the query workload. DPC sees only this public metadata:

metadata:
  name: lineage-reader
  namespace: products
  annotations:
    data.devantler.tech/arango-publication: v1
    data.devantler.tech/arango-user: catalog-reader
    data.devantler.tech/arango-database: catalog
    data.devantler.tech/arango-graph: lineage
    data.devantler.tech/arango-access: read-only
    data.devantler.tech/arango-collections: products,relations
  ownerReferences:
    - apiVersion: database.arangodb.com/v1
      kind: ArangoDeployment
      name: lineage
      uid: <current-source-uid>

Under v1, read-only declares this grant profile, including no other positive collection grants and explicit denials for unpublished application collections. This is publisher intent, not proof that future collections are automatically isolated. See the upstream permission resolution rules. Identifiers start with an ASCII letter followed by at most 63 ASCII letters, digits, underscores or hyphens. Collections form a comma-separated list of 1–64 unique identifiers without whitespace or wildcards. Root, operator, internal and backup users are rejected in any letter case. System names, unsupported versions and writable declarations are rejected. The selected Secret must match the current source owner. Default/configured JWT, root-password and operator credential publications are unsupported. Do not relabel operator Secrets as application publications.

Apply the Graph observer Role, adjusting its ServiceAccount binding. It permits only exact-name GETs on lineage and lineage-reader in products. Kubernetes authorizes a whole-Secret GET despite metadata negotiation.

These checks establish operator-reported current-spec readiness and publisher intent. They do not verify the installed operator binary, immutable running image, credential values, effective permissions, graph existence, queries, backups or distribution support. ArangoDB's Community binary terms restrict deployment uses; this adapter neither deploys nor licenses the database. Independently verify the applicable edition and terms. The required real Graph acceptance separately exercises the pinned operator and authenticated traversal; its current-head run must pass.

Lifecycle and rollout

Creation and deletion stay with the operator. A missing source reports SourceNotFound; deleting one reports SourceDeleting. Recreating it changes its UID, so an old Secret cannot satisfy publication ownership. Credential rotation at the same Secret name does not require a DataProduct change or copy values into status. Removing the source declaration or deleting the DataProduct leaves the source and Secret independently owned; the controller adds no finalizers or owner references.

The hosted source-observation suite exercises admission, scoped permissions, both gates, readiness loss/recovery, publication ownership and retention using synthetic SQL, Document and Graph status fixtures. It does not install database operators or prove database availability. The separate required real Document acceptance installs Percona and exercises authenticated queries, effective privileges, rotation, outage recovery and retained data. Its current-head run must pass; synthetic observer results cannot replace that evidence. The required real Graph acceptance applies the same boundary to ArangoDB traversal, effective grants and retained source recovery. The remaining real provider matrix is tracked in #38, and released deployment acceptance is required before retiring the gate in #128.