feat(datasource,runtime): teardown through the one datasource path, honoring adopted pools (#3993); ADR-0062 status flip (#3992)

.changeset/datasource-teardown-ownership.md

@@ -0,0 +1,44 @@
+---
+"@objectstack/service-datasource": minor
+"@objectstack/runtime": minor
+---
+
+feat(datasource,runtime): kernel teardown disconnects through the one datasource path — and never closes an adopted pool (#3993)
+
+After the #3826 connect convergence, ADR-0062 D5's "owns connect/disconnect"
+was half-true: nothing disconnected the `default` (or a declared datasource's
+pool) on graceful shutdown. `DriverPlugin` never had teardown, `ObjectQLPlugin`
+teardown never touched drivers, and the kernel's actual teardown phase is
+`destroy()` — the Plugin contract has no `stop()`, so stray `stop` methods were
+never called by anything.
+
+The disconnect half now mirrors the connect half:
+
+- **`DatasourceConnectionService.disconnect(name, { asDefault })`** resolves
+  the default under its NATURAL name (the same #3826 rule that makes
+  `drivers.get('default')` impossible — the old lookup could never have found
+  it), and honours a new ownership discriminator recorded at connect time.
+- **`disconnectAll()`** closes exactly the pools THIS service opened —
+  `'connected'` states only. `already-registered` drivers belong to whoever
+  registered them (an `onEnable` bridge, the default's idempotent replay) and
+  are never touched.
+- **`DatasourceDriverHandle.ownership: 'factory' | 'host'`** is the
+  discriminator. `createPrebuiltDriverFactory` stamps its handles `'host'`:
+  an ADOPTED instance's pool outlives the kernel (the cloud control-plane
+  driver doubles as every environment kernel's proxy base; per-environment
+  drivers are registry-cached across kernel rebuilds), so kernel teardown —
+  including a cloud LRU eviction's `kernel.shutdown()` — clears the retained
+  verdict but NEVER closes the pool. Factory-built instances disconnect as
+  before there was a before.
+- **`DefaultDatasourcePlugin.destroy()`** and
+  **`DatasourceAdminServicePlugin.destroy()`** wire the sweep at the kernel's
+  real teardown phase, best-effort (a failed disconnect never masks shutdown).
+
+A welcome side effect: a file-backed `sqlite-wasm` default with
+`persist: 'on-disconnect'` now actually flushes on graceful shutdown.
+
+Also flips ADR-0062's status to reflect the completed convergence (#3992):
+D1 is fully implemented across both repos since cloud#915; the remaining
+`DriverPlugin` uses are documented named-auxiliary/escape-hatch cases, and the
+degraded-boot parity guard stays with its role shifted to "the escape hatches
+must not drift".

docs/adr/0062-external-datasource-runtime.md

@@ -1,6 +1,6 @@
 # ADR-0062: External Datasource Runtime — connection lifecycle, credentials, visibility & query completeness
 
-**Status**: Accepted (2026-06-22) — D2–D8 implemented (`service-datasource` connection service + opt-in-safe gate + fail-closed connect policy; native-SQL declines external per D6; D7 lint in `validate-expressions.ts`). **D1 implemented for every open-core boot path** (#3826: the standalone/artifact `default` and the CLI serve fallback are declared definitions connected through the one service — see the status notes under D1); the cloud repo's own stack composition is the last remaining second site, and the parity guard stays until it converges.
+**Status**: Accepted (2026-06-22) — D1–D8 implemented. **D1 fully implemented across both repos** (#3826, closed): every open-core boot path (#3968) and every cloud composition (cloud#915 — control-plane preset, objectos `artifact-kernel-factory`, `DefaultEnvironmentKernelFactory`) connects its `default` through the one `DatasourceConnectionService` path; the remaining `DriverPlugin` uses are documented named-auxiliary/escape-hatch cases, and the degraded-boot parity guard stays to pin them (see the status notes under D1). D5's teardown half landed with #3993: kernel teardown disconnects through the same service, honouring adopted (host-owned) instances — see the amendment under D5.
 
 **Supersedes the runtime portions of**: ADR-0015 §18 addendum (kept as the historical record). ADR-0015 remains the canonical spec/binding decision; this ADR is the canonical *runtime* decision.
 
@@ -68,7 +68,7 @@ Introduce a single service that, given a datasource definition, builds a driver
 >
 > **Config-load fallback converged too (#3826, third pass).** `createStorageDriver` is gone: the CLI's serve fallback (a host `objectstack.config.ts` with objects but no driver plugin) now emits a definition via `resolveStorageDefinition` and hands it to the same `DefaultDatasourcePlugin`. `mysql` joined the shared factory for it; the dev loosen-only self-heal (#2186) rides as `config.autoMigrate` and the CLI's wasm persistence mode as `config.persist` — host-composition passthroughs the factory honours, never part of the app-facing datasource spec. `turso`/libSQL keeps its loud typed failure at *resolution* (nothing is constructed to fail later). The `telemetry` sibling datasource deliberately stays a pre-built `DriverPlugin` (the documented escape hatch for named auxiliary drivers): it is best-effort, dev-oriented, and its own `resolveSqliteDriver` step-down check replaces the old primary-resolution coupling.
 >
-> **Remaining second site: the cloud repo's own compositions** — the real primary-driver sites are `environment-kernel-factory.ts` (each environment's per-tenant driver) and the control-plane preset fed by `cloud-stack.ts`'s `buildControlDriver`, which also owns the `turso` driver. Until they converge, `packages/runtime/src/degraded-boot-parity.test.ts` remains load-bearing: it pins both connect paths to the same operator-visible contract (fail-fast by default, identical `OS_ALLOW_DRIVER_CONNECT_FAILURE` parsing, `DEGRADED BOOT` on stderr). #3741 → #3758 was exactly the miss it exists to catch.
+> **Convergence completed — the cloud compositions landed (cloud#915), no second site remains.** The cloud repo's three primary-driver sites — the control-plane preset (fed by `cloud-stack.ts`'s `buildControlDriver`, whose instance doubles as every environment kernel's proxy base), `artifact-kernel-factory.ts` (the objectos per-tenant hot path), and both `DefaultEnvironmentKernelFactory` paths — now boot through `DefaultDatasourcePlugin` with their pre-built instances adopted via `createPrebuiltDriverFactory`. Every remaining `DriverPlugin` use is a **documented named-auxiliary / escape-hatch case**, not a default path: the framework's `telemetry` sibling, the cloud proxy `cloud` datasource (`registerAsDefault: false`), the objectos host routing shell, the artifact factory's `'cloud'` alias metadata registration, and test injection. `packages/runtime/src/degraded-boot-parity.test.ts` **stays load-bearing after the convergence**, with its role shifted from "the unconverged second implementation must not drift" to "the escape hatches must not drift": as long as `ObjectQLEngine.init()` can throw a connect verdict at all (the boot re-verification, pre-built `DriverPlugin` drivers), that verdict must match the service's. #3741 → #3758 was exactly the miss it exists to catch.
 >
 > **The convergence seam (#3826, fourth pass).** Two properties of the cloud composition made "just declare it" impossible: its driver kinds live outside open-core (`turso`), and its instances are *pooled beyond one kernel* (the control-plane driver doubles as the proxy base of every environment kernel; per-environment drivers are cached across kernel rebuilds — reconstruction per boot would multiply pools). So `DefaultDatasourcePlugin` now accepts an **injected `IDatasourceDriverFactory`** (defaulting to the shared open-core factory), and `createPrebuiltDriverFactory` wraps an already-built instance as a factory — the "adopt an existing driver" entry point this ADR's first pass found missing, landed *as a factory* so the connect orchestration (policy-free init connect, `bootCritical` verdict, shared escape hatch, start() replay into retained state) stays this one implementation. Construction and pooling remain host concerns; only the verdict converges. The `@objectstack/verify` dogfood harness also boots through the declared default now (not the `DriverPlugin` escape hatch), making the §Risk mitigation — "behind the dogfood gate" — actually true for the converged path.
 
@@ -108,6 +108,8 @@ Code-defined datasources surface in `GET /api/v1/datasources`, `GET /api/v1/meta
 >
 > **A `DatasourceConnectPolicy` denial is not a connect failure** and stays metadata-only, unchanged. A multi-tenant host that blocks egress for a tenant's plan is making a deliberate decision about a datasource it knows it is refusing; fail-fasting there would turn a policy verdict into a boot outage for every tenant on the shared runtime. The fail-fast set covers *failures* — unreachable, unauthenticated, unsupported — not *refusals*.
 
+> **Amendment (#3993) — teardown is one implementation too, and it distinguishes owned from adopted.** After the D1 connect convergence, "owns connect/disconnect" was half-true: nothing disconnected the `default` (or a declared datasource's pool) on graceful shutdown — `DriverPlugin` never had teardown, `ObjectQLPlugin`'s teardown never touched drivers, and the kernel's actual teardown phase is **`destroy()`** (the Plugin contract has no `stop()`; stray `stop` methods were never called). Now `DatasourceConnectionService` owns the disconnect half symmetrically: `disconnect(name, { asDefault })` resolves the default under its **natural name** (the same #3826 rule that makes `drivers.get('default')` impossible), and `disconnectAll()` — wired from `DefaultDatasourcePlugin.destroy()` and `DatasourceAdminServicePlugin.destroy()` — closes **exactly the pools this service opened** (`'connected'` states; `already-registered` drivers belong to whoever registered them). The factory handle carries the discriminator: **`ownership: 'host'`** (set by `createPrebuiltDriverFactory`) marks an ADOPTED instance whose pool outlives the kernel — the cloud constraint: the control-plane driver doubles as every environment kernel's proxy base, per-environment drivers are registry-cached across kernel rebuilds, and an LRU eviction (`kernel.shutdown()`) closing a shared pool would pull it from under every other consumer. For those, teardown clears the retained verdict and leaves the pool to its host. A welcome side effect: a file-backed `sqlite-wasm` default with `persist: 'on-disconnect'` now actually flushes on graceful shutdown.
+
 ### D6 — Native-analytics SQL honors the remote table/columns
 
 The analytics native-SQL strategy compiles its own `FROM "<table>"` / column references outside the driver (ADR-0015 §18 noted this). It must resolve an external object's physical table (`remoteName`/`remoteSchema`) and columns (`columnMap`) the same way `SqlDriver` now does — reusing the driver's resolution (e.g. an exposed `physicalTableFor(object)` / `physicalColumnFor(object, field)`), not a second copy. Until then, analytics over external objects stays disabled rather than silently querying the wrong table.

packages/runtime/src/default-datasource-plugin.test.ts

@@ -194,6 +194,37 @@ describe('DefaultDatasourcePlugin — the default datasource as a declaration (#
     try { await (kernel as any)?.stop?.(); } catch { /* noop */ }
   }, BOOT_TIMEOUT);
 
+  it('kernel teardown disconnects an OWNED (factory-built) default through the one service (#3993)', async () => {
+    const kernel = await assemble({});
+    await kernel.bootstrap();
+    const engine = kernel.getService<any>('data');
+    const drv = engine.getDriverByName(engine.getDefaultDriverName());
+    let disconnects = 0;
+    const orig = drv.disconnect?.bind(drv);
+    drv.disconnect = async () => { disconnects += 1; return orig?.(); };
+    await (kernel as any).shutdown();
+    expect(disconnects).toBeGreaterThan(0);
+  }, BOOT_TIMEOUT);
+
+  it('kernel teardown NEVER closes an ADOPTED (host-owned) default — the pool outlives the kernel (#3993)', async () => {
+    // The cloud shape: the instance is shared beyond this kernel (proxy base /
+    // registry cache). An LRU eviction shutting the kernel down must not pull
+    // the pool from under every other consumer.
+    const { createPrebuiltDriverFactory } = await import('@objectstack/service-datasource');
+    const { InMemoryDriver } = await import('@objectstack/driver-memory');
+    const hostBuilt = new InMemoryDriver() as any;
+    let disconnects = 0;
+    const orig = hostBuilt.disconnect?.bind(hostBuilt);
+    hostBuilt.disconnect = async () => { disconnects += 1; return orig?.(); };
+    const kernel = await assemble({
+      driver: 'turso',
+      factory: createPrebuiltDriverFactory(hostBuilt, { driverId: 'turso' }),
+    });
+    await kernel.bootstrap();
+    await (kernel as any).shutdown();
+    expect(disconnects).toBe(0);
+  }, BOOT_TIMEOUT);
+
   it("rejects an app bundle that declares a datasource named 'default' (host-reserved name)", async () => {
     const kernel = await assemble({
       bundle: {

packages/runtime/src/default-datasource-plugin.ts

@@ -98,6 +98,8 @@ export class DefaultDatasourcePlugin implements Plugin {
   private readonly def: DefaultDatasourceDefinition;
   private readonly dev?: boolean;
   private readonly factory?: IDatasourceDriverFactory;
+  /** The init()-time local connection service — held for destroy()'s teardown. */
+  private connection?: DatasourceConnectionService;
 
   constructor(def: DefaultDatasourceDefinition, opts: DefaultDatasourcePluginOptions = {}) {
     this.def = def;
@@ -131,6 +133,7 @@ export class DefaultDatasourcePlugin implements Plugin {
       // packages.
       logger: ctx.logger as unknown as ConstructorParameters<typeof DatasourceConnectionService>[0]['logger'],
     });
+    this.connection = connection;
 
     // Throws on failure (bootCritical ⇒ fail-fast per ADR-0062 D5), aborting
     // bootstrap exactly like the engine-level guard did — same escape hatch,
@@ -243,4 +246,24 @@ export class DefaultDatasourcePlugin implements Plugin {
       ctx.logger.debug('[DefaultDatasourcePlugin] metadata service unavailable — default not listed', { error: e });
     }
   }
+
+  /**
+   * Kernel teardown (ADR-0062 D5, #3993): the default disconnects through the
+   * SAME service that connected it — one teardown implementation, mirroring
+   * the one connect implementation. The service resolves the driver the way
+   * the default was registered (natural name via `asDefault`) and honours
+   * ownership: a host-owned ADOPTED instance (`createPrebuiltDriverFactory`,
+   * the cloud compositions — pools shared beyond this kernel) is never
+   * closed here; only the retained verdict is cleared. Note the kernel's
+   * teardown phase is `destroy()` — `stop()` exists nowhere in the Plugin
+   * contract and is never called.
+   */
+  destroy = async () => {
+    try {
+      await this.connection?.disconnect('default', { asDefault: true });
+    } catch {
+      // Teardown is best-effort — the kernel is going away either way, and a
+      // failed disconnect must not mask the real shutdown path.
+    }
+  };
 }

packages/services/service-datasource/src/__tests__/prebuilt-driver-factory.test.ts

@@ -112,6 +112,80 @@ describe('createPrebuiltDriverFactory — through DatasourceConnectionService (t
     expect(driver.calls).toContain('connect');
   });
 
+  it("stamps the handle 'host'-owned, and kernel teardown leaves the adopted pool alone (#3993)", async () => {
+    // The cloud constraint: the adopted instance's pool outlives this kernel
+    // (proxy base / registry cache). disconnect() must clear the retained
+    // verdict WITHOUT closing the pool.
+    const driver = stubDriver('turso');
+    const factory = createPrebuiltDriverFactory(driver, { driverId: 'turso' });
+    const handle: any = await factory.create({ driver: 'turso', config: {} });
+    expect(handle.ownership).toBe('host');
+
+    const { svc } = service(factory);
+    const result = await svc.connect(
+      { name: 'default', driver: 'turso', config: {}, origin: 'code', bootCritical: true },
+      { asDefault: true, context: { origin: 'code', trigger: 'declared-auto' } },
+    );
+    expect(result.ownership).toBe('host');
+    expect(svc.getConnectionState('default')?.ownership).toBe('host');
+
+    await svc.disconnect('default', { asDefault: true });
+    expect(driver.calls).not.toContain('disconnect'); // the pool belongs to the host
+    expect(svc.getConnectionState('default')).toBeUndefined(); // the verdict is gone
+  });
+
+  it('a FACTORY-built default IS disconnected at teardown — natural-name resolution via asDefault (#3993)', async () => {
+    // Without ownership the instance was built for this connect: teardown may
+    // close it. `asDefault` must resolve the driver under its natural name —
+    // `getDriverByName('default')` can never find it (#3826).
+    const driver = stubDriver('sql');
+    const factoryBuilt = {
+      supports: () => true,
+      create: () => ({
+        connect: async () => { await driver.connect(); },
+        disconnect: async () => { await driver.disconnect(); },
+        driver,
+      }),
+    };
+    const { svc } = service(factoryBuilt as any);
+    await svc.connect(
+      { name: 'default', driver: 'sqlite', config: {}, origin: 'code', bootCritical: true },
+      { asDefault: true, context: { origin: 'code', trigger: 'declared-auto' } },
+    );
+    await svc.disconnect('default', { asDefault: true });
+    expect(driver.calls).toContain('disconnect');
+    expect(svc.getConnectionState('default')).toBeUndefined();
+  });
+
+  it("disconnectAll() closes only what THIS service opened — 'connected' states, never 'already-registered' (#3993)", async () => {
+    const opened = stubDriver('sql');
+    const factoryBuilt = {
+      supports: () => true,
+      create: () => ({ disconnect: async () => { await opened.disconnect(); }, driver: opened }),
+    };
+    const { svc, drivers } = service(factoryBuilt as any);
+    // A driver someone ELSE registered (the D8 onEnable escape hatch): the
+    // idempotency guard records `already-registered` — not ours to close.
+    const foreign = stubDriver('warehouse');
+    drivers.set('warehouse', foreign);
+    const pre = await svc.connect(
+      { name: 'warehouse', driver: 'sqlite', config: {}, autoConnect: true },
+      { context: { origin: 'code', trigger: 'declared-auto' } },
+    );
+    expect(pre.status).toBe('already-registered');
+    // A pool this service opened.
+    await svc.connect(
+      { name: 'analytics', driver: 'sqlite', config: {}, autoConnect: true },
+      { context: { origin: 'code', trigger: 'declared-auto' } },
+    );
+
+    await svc.disconnectAll();
+    expect(opened.calls).toContain('disconnect');
+    expect(foreign.calls).not.toContain('disconnect');
+    expect(svc.getConnectionState('analytics')).toBeUndefined();
+    expect(svc.getConnectionState('warehouse')).toBeDefined(); // untouched
+  });
+
   it('a failing adopted instance takes the SAME bootCritical fail-fast verdict', async () => {
     const driver = stubDriver('turso', { failConnect: true });
     const { svc } = service(createPrebuiltDriverFactory(driver, { driverId: 'turso' }));

packages/services/service-datasource/src/contracts/datasource-driver-factory.ts

@@ -37,6 +37,19 @@ export interface DatasourceConnectionSpec {
   pool?: Record<string, unknown>;
 }
 
+/**
+ * Who owns the TEARDOWN of the driver instance behind a handle (ADR-0062 D5,
+ * #3993):
+ *  - `'factory'` (the default when absent) — the factory built this instance
+ *    for this connect; the connection service may disconnect it when the
+ *    kernel tears down.
+ *  - `'host'` — an adopted, pre-built instance (`createPrebuiltDriverFactory`)
+ *    whose lifecycle outlives the kernel (a pool shared across environment
+ *    kernels, a control-plane driver doubling as a proxy base). Kernel
+ *    teardown must NEVER disconnect it; the host does, on its own schedule.
+ */
+export type DatasourceDriverOwnership = 'factory' | 'host';
+
 /**
  * A live (or lazily-connecting) driver handle. Intentionally structural and
  * fully optional so any concrete driver satisfies it — the admin service uses
@@ -47,6 +60,8 @@ export interface DatasourceDriverHandle {
   connect?(): Promise<void>;
   /** Close the connection / pool. */
   disconnect?(): Promise<void>;
+  /** Teardown ownership of the underlying instance — see {@link DatasourceDriverOwnership}. */
+  ownership?: DatasourceDriverOwnership;
   /** Cheap liveness round-trip (preferred for probes). */
   ping?(): Promise<unknown>;
   /** Introspect the live schema (fallback probe when `ping` is absent). */