fix(driver-sql): reclaimSpace() returns the freed bytes from the SQLite -wal sidecar too, never waiting on another connection (#20426)

.changeset/20106-reclaim-space-full-freelist.md

@@ -1,3 +1,3 @@
 ---
 '@objectstack/driver-sql': patch
 '@objectstack/driver-turso': patch
@@ -14,6 +14,4 @@
 
 `SqliteWasmDriver` was already complete: its dialect steps every PRAGMA to the end (300 → 0 before and after this change).
 
-On a file-backed database in WAL mode (the default) the database file shrinks once a checkpoint runs, and during the call the freed pages pass through the `-wal` file, which keeps its size until the last connection closes.
-
 Nothing to migrate: `reclaimSpace()` keeps its signature, and a database whose `auto_vacuum` mode is not `INCREMENTAL` still reclaims nothing, as before.

.changeset/20426-reclaim-space-wal-sidecar.md

@@ -0,0 +1,15 @@
+---
+'@objectstack/driver-sql': patch
+---
+
+fix(driver-sql): `reclaimSpace()` returns the freed bytes from the SQLite `-wal` sidecar too, and never waits on another connection (#20426)
+
+Clause-②: no
+
+On a file-backed SQLite database in WAL mode, the default, `reclaimSpace()` returned the whole freelist but left the freed bytes in the `-wal` sidecar. At 25,754 free pages the database file went from 103,149,568 to 16,384 bytes while the `-wal` file went from 4,255,992 to 94,430,432 bytes, and it kept that size until the last connection closed. The lifecycle sweep calls this method after every sweep that deleted rows, and it reported the datasource as reclaimed.
+
+On better-sqlite3 (`SqlDriver`, and `TursoDriver` in local mode) the vacuum now runs in chunks of a quarter of the connection's page cache, 1,000 pages at the default cache size, with a `PASSIVE` checkpoint after each chunk. One `TRUNCATE` checkpoint closes the call, taken with a busy timeout of 0, so it never waits on another connection. On the same database, file plus `-wal` goes from 107,405,560 to 16,384 bytes while the driver is still open.
+
+When another connection holds a read transaction, the call still returns without waiting (47 to 66 ms measured; a `TRUNCATE` checkpoint that waits blocked the process for the connection's 5-second busy timeout). The pages are off the freelist, and their bytes leave the files at a later checkpoint. The call no longer grows the pair either: 107,405,560 bytes before and after, where the single statement grew it to 197,580,000.
+
+A database in rollback-journal (`delete`) mode behaves as before. The remote `TursoDriver` route and `SqliteWasmDriver` are unchanged. Nothing to migrate: `reclaimSpace()` keeps its signature.

packages/drivers/driver-sql/src/sql-driver-sqlite-reclaim-space.test.ts

@@ -8,9 +8,16 @@
 // `Statement.run()`, which steps it once, so this method used to free ONE page
 // per call: 300 → 299 from a second connection, and the lifecycle sweep that
 // calls it after every bulk delete left the file at its high-water mark.
+//
+// In WAL mode — the file-backed default — the freed bytes must leave the
+// `-wal` sidecar too, so every size below is the database file PLUS the
+// `-wal` file, read while the driver is still open. One statement over the
+// whole freelist spilled its pages into the WAL (25,754 free pages: the file
+// went to 16,384 bytes and the `-wal` to 94,430,432, held until the last
+// connection closed), and nothing checkpointed or truncated it.
 
 import { afterEach, describe, expect, it } from 'vitest';
-import { mkdtempSync, rmSync, statSync } from 'node:fs';
+import { existsSync, mkdtempSync, rmSync, statSync } from 'node:fs';
 import { tmpdir } from 'node:os';
 import { join } from 'node:path';
 import knex, { type Knex } from 'knex';
@@ -28,6 +35,16 @@ function tempDb(): string {
   return join(dir, 'app.db');
 }
 
+/** The `-wal` sidecar's size in bytes; 0 when there is none. */
+function walBytes(filename: string): number {
+  return existsSync(`${filename}-wal`) ? statSync(`${filename}-wal`).size : 0;
+}
+
+/** The database file and its `-wal` sidecar, as the file system reports them. */
+function onDisk(filename: string): { file: number; wal: number } {
+  return { file: statSync(filename).size, wal: walBytes(filename) };
+}
+
 /** A connected driver on `filename`, and a `close()` the cleanup then skips. */
 async function openDriver(
   filename: string,
@@ -73,19 +90,92 @@ async function secondConnection(filename: string): Promise<{ freelist: number; p
 }
 
 describe('SqlDriver.reclaimSpace() on better-sqlite3 returns the whole freelist', () => {
-  it('WAL (the file-backed default): every free page leaves the database, and the file shrinks once closed', async () => {
+  it('WAL (the file-backed default): every free page leaves the database, and its bytes leave the -wal sidecar too, while the driver is open', async () => {
     const file = tempDb();
     const { driver, close } = await openDriver(file);
     await freePages(driver, 300);
     const before = await secondConnection(file);
     expect(before.freelist).toBeGreaterThanOrEqual(250);
+    expect(walBytes(file)).toBeGreaterThan(0);
 
     await driver.reclaimSpace();
 
     const after = await secondConnection(file);
     expect(after).toEqual({ freelist: 0, pages: before.pages - before.freelist });
+    expect(onDisk(file)).toEqual({ file: after.pages * PAGE_SIZE, wal: 0 });
     await close();
-    expect(statSync(file).size).toBe(after.pages * PAGE_SIZE);
+    expect(onDisk(file)).toEqual({ file: after.pages * PAGE_SIZE, wal: 0 });
+  });
+
+  it('WAL, another connection holding a read transaction: the call neither waits nor fails, and writes the WAL a chunk at a time', async () => {
+    const file = tempDb();
+    // Fill and delete, then reopen: the last close checkpoints and removes the
+    // WAL, so with the reader below pinning every frame, the `-wal` size after
+    // the call is exactly what the call wrote.
+    const first = await openDriver(file);
+    await freePages(first.driver, 600);
+    await first.close();
+    const { driver } = await openDriver(file);
+    // A page cache of about 100 pages (400 KiB): one statement over 600 free
+    // pages outgrows it and spills them into the WAL, and so does any fixed
+    // chunk sized for the default cache.
+    await driver.execute('PRAGMA cache_size = -400');
+
+    const reader: Knex = knex({ client: 'better-sqlite3', connection: { filename: file }, useNullAsDefault: true });
+    cleanup.push(() => reader.destroy());
+    const snapshot = await reader.transaction();
+    // Runs before the destroy above: a failed assertion must not leave the
+    // reader's connection checked out, or the destroy waits for it.
+    cleanup.push(async () => {
+      if (!snapshot.isCompleted()) await snapshot.rollback();
+    });
+    await snapshot.raw('SELECT count(*) AS n FROM bulk');
+    const before = await secondConnection(file);
+    expect(before.freelist).toBeGreaterThanOrEqual(550);
+    const walBefore = walBytes(file);
+
+    const started = performance.now();
+    await expect(driver.reclaimSpace()).resolves.toBeUndefined();
+    const elapsed = performance.now() - started;
+
+    // Waiting on the reader would take the connection's whole busy timeout.
+    const [{ timeout }] = (await driver.execute('PRAGMA busy_timeout')) as Array<{ timeout: number }>;
+    expect(timeout).toBe(5000);
+    expect(elapsed).toBeLessThan(timeout / 2);
+    expect(await secondConnection(file)).toEqual({ freelist: 0, pages: before.pages - before.freelist });
+    // Measured: 0.08 of the freed bytes in 25-page chunks, 0.87 for one
+    // statement and for one 1,000-page chunk alike.
+    expect(walBytes(file) - walBefore).toBeLessThan((before.freelist * PAGE_SIZE) / 4);
+
+    // Once the reader is gone, the next reclaim returns what this one left.
+    await snapshot.commit();
+    await freePages(driver, 10);
+    expect((await secondConnection(file)).freelist).toBeGreaterThan(0);
+    await driver.reclaimSpace();
+    const settled = await secondConnection(file);
+    expect(settled.freelist).toBe(0);
+    expect(onDisk(file)).toEqual({ file: settled.pages * PAGE_SIZE, wal: 0 });
+  });
+
+  it('control: a file whose auto_vacuum is still NONE frees nothing, and the loop stops', async () => {
+    const file = tempDb();
+    // Created before the driver's INCREMENTAL default: a table already exists,
+    // so that default cannot change this file's layout.
+    const legacy: Knex = knex({ client: 'better-sqlite3', connection: { filename: file }, useNullAsDefault: true });
+    await legacy.raw('CREATE TABLE legacy_marker (x INTEGER)');
+    await legacy.destroy();
+    const { driver } = await openDriver(file);
+    await freePages(driver, 300);
+    const before = await secondConnection(file);
+    expect(before.freelist).toBeGreaterThanOrEqual(250);
+    const diskBefore = onDisk(file);
+
+    await expect(driver.reclaimSpace()).resolves.toBeUndefined();
+
+    expect(await secondConnection(file)).toEqual(before);
+    const disk = onDisk(file);
+    expect(disk.file).toBe(before.pages * PAGE_SIZE);
+    expect(disk.file + disk.wal).toBeLessThanOrEqual(diskBefore.file + diskBefore.wal);
   });
 
   it('DELETE journal: the file shrinks while the driver is still open', async () => {
@@ -100,19 +190,21 @@ describe('SqlDriver.reclaimSpace() on better-sqlite3 returns the whole freelist'
 
     const after = await secondConnection(file);
     expect(after).toEqual({ freelist: 0, pages: before.pages - before.freelist });
-    expect(statSync(file).size).toBe(after.pages * PAGE_SIZE);
+    expect(onDisk(file)).toEqual({ file: after.pages * PAGE_SIZE, wal: 0 });
   });
 
-  it('control: an empty freelist resolves, and nothing changes', async () => {
+  it.each(['wal', 'delete'] as const)('control, %s journal: an empty freelist resolves, and nothing changes', async (mode) => {
     const file = tempDb();
-    const { driver } = await openDriver(file, { sqliteJournalMode: 'delete' });
+    const { driver } = await openDriver(file, { sqliteJournalMode: mode });
     await freePages(driver, 0);
     const before = await secondConnection(file);
     expect(before.freelist).toBe(0);
+    const diskBefore = onDisk(file);
 
     await expect(driver.reclaimSpace()).resolves.toBeUndefined();
 
     expect(await secondConnection(file)).toEqual(before);
+    expect(onDisk(file)).toEqual(diskBefore);
   });
 
   it('the pooled connection is handed back: the driver answers a query after the call', async () => {

packages/drivers/driver-sql/src/sql-driver.ts

@@ -5440,6 +5440,79 @@ function formatDuplicateGroups(duplicates: ReadonlyArray<{ key: string; rows: nu
   return duplicates.length > 5 ? `${shown}; \u2026and ${duplicates.length - 5} more group(s)` : shown;
 }
 
+/** The part of a better-sqlite3 `Database` that {@link reclaimBetterSqlite3} drives. */
+interface BetterSqlite3Connection {
+  exec(sql: string): unknown;
+  pragma(source: string, options?: { simple: boolean }): unknown;
+}
+
+/**
+ * The better-sqlite3 arm of `SqlDriver.reclaimSpace`: return the whole freelist,
+ * and return the bytes it passes through the `-wal` sidecar too, without ever
+ * waiting on another connection. Every statement runs through the binding's
+ * `exec()` / `pragma()`, which step to completion. Synchronous by design:
+ * nothing else runs on the connection between the chunks, so the freelist
+ * only shrinks while the loop runs.
+ *
+ * Why chunks. In WAL mode one `PRAGMA incremental_vacuum` over a large
+ * freelist is one transaction whose dirty pages outgrow the page cache, so
+ * SQLite spills them into the WAL before the commit truncates them away.
+ * Measured at 25,754 free pages: the database file went to 16,384 bytes and
+ * the `-wal` sidecar to 94,430,432, held until the last connection closed. A
+ * chunk that stays inside the page cache writes only the pages its commit
+ * keeps. The chunk is a quarter of this connection's page cache — 1,000 pages
+ * at better-sqlite3's default `cache_size` (-16000 KiB) and 4 KiB pages.
+ * Readings with a reader pinning the WAL, so every frame written stays
+ * visible: 2,000-page chunks left 883 frames, 4,000 left 3,779, and one
+ * statement 22,920; with a 2 MB cache, 250-page chunks left 1,121 and
+ * 1,000-page chunks 15,491.
+ *
+ * Why these two checkpoints. A `PASSIVE` checkpoint after each chunk moves
+ * its frames into the database and lets the next chunk restart the WAL from
+ * its start; it never waits. What it cannot do is shrink the sidecar, which
+ * keeps its high-water size until something truncates it. So one
+ * `TRUNCATE` checkpoint closes the call, under a busy timeout of 0 for that
+ * one statement and the connection's own timeout put back afterwards: a
+ * `TRUNCATE` checkpoint waits for other connections' readers through the
+ * busy handler, and on this synchronous binding that wait blocks the whole
+ * process — measured at 5,333 ms against a reader in the same process, the
+ * connection's 5,000 ms timeout. When another connection is reading, the
+ * `PASSIVE` checkpoints move only the frames that reader no longer needs and
+ * the `TRUNCATE` one answers "busy" as a result row, not as an error: the
+ * pages are off the freelist, and their bytes leave the files at a later
+ * checkpoint (the next call, SQLite's own auto-checkpoint, or the last
+ * connection closing). Outside WAL mode both checkpoints are no-ops.
+ *
+ * The loop stops when the freelist is empty or a chunk frees nothing: a file
+ * whose `auto_vacuum` is still `NONE` never shrinks its freelist through
+ * this statement (one full `VACUUM` adopts INCREMENTAL, see `SqlDriver.connect`).
+ *
+ * Module-local, like {@link formatDuplicateGroups}: `SqlDriver`'s `.d.ts`
+ * carries its non-public members too, and this helper is no entry point.
+ */
+function reclaimBetterSqlite3(db: BetterSqlite3Connection): void {
+  const scalar = (pragma: string): number => Number(db.pragma(pragma, { simple: true }));
+  let free = scalar('freelist_count');
+  if (free === 0) return;
+  const cacheSize = scalar('cache_size');
+  const cachePages = cacheSize >= 0 ? cacheSize : Math.floor((-cacheSize * 1024) / scalar('page_size'));
+  const chunk = Math.max(1, Math.floor(cachePages / 4));
+  for (;;) {
+    db.exec(`PRAGMA incremental_vacuum(${chunk})`);
+    db.exec('PRAGMA wal_checkpoint(PASSIVE)');
+    const left = scalar('freelist_count');
+    if (left === 0 || left >= free) break;
+    free = left;
+  }
+  const busyTimeout = scalar('busy_timeout');
+  db.pragma('busy_timeout = 0');
+  try {
+    db.exec('PRAGMA wal_checkpoint(TRUNCATE)');
+  } finally {
+    db.pragma(`busy_timeout = ${busyTimeout}`);
+  }
+}
+
 export class SqlDriver implements IDataDriver {
   // IDataDriver metadata
   public readonly name: string = 'com.objectstack.driver.sql';
@@ -10944,14 +11017,18 @@ export class SqlDriver implements IDataDriver {
    * `knex.raw` below). knex's node-sqlite3 client runs a raw statement with
    * `Database.all()`, which reads every row too (read from knex's source; that
    * binding is not installed in this repository).
+   *
+   * On better-sqlite3 the freed bytes also leave the `-wal` sidecar, which is
+   * what a file-backed database in WAL mode (the default, see
+   * {@link applySqliteJournalMode}) needs — see `reclaimBetterSqlite3`.
    */
   async reclaimSpace(_options?: DriverOptions): Promise<void> {
     if (!this.isSqlite) return;
     const client = this.knex.client;
     if (client.driverName === 'better-sqlite3') {
       const connection = await client.acquireConnection();
       try {
-        connection.exec('PRAGMA incremental_vacuum');
+        reclaimBetterSqlite3(connection);
       } finally {
         await client.releaseConnection(connection);
       }

packages/drivers/driver-turso/src/turso-remote-inherited-members.test.ts

@@ -39,7 +39,7 @@
  *    | the other 13 | unchanged, and true on this face | unchanged |
  */
 
-import { mkdtempSync, rmSync, statSync } from 'node:fs';
+import { existsSync, mkdtempSync, rmSync, statSync } from 'node:fs';
 import { tmpdir } from 'node:os';
 import { join } from 'node:path';
 import { afterAll, afterEach, describe, expect, it } from 'vitest';
@@ -306,6 +306,17 @@ describe('reclaimSpace(): the local statement, run to completion on the remote d
     expect((await pageState(file)).freelist).toBe(0);
   });
 
+  it('local face: in WAL mode the freed bytes leave the -wal sidecar too, while the driver is still open', async () => {
+    const { driver, file } = await withFreePages('local');
+    const wal = () => (existsSync(`${file}-wal`) ? statSync(`${file}-wal`).size : 0);
+    const pageSize = await secondConnection(file, 'PRAGMA page_size');
+    expect(wal()).toBeGreaterThan(0);
+    await driver.reclaimSpace();
+    const after = await pageState(file);
+    expect(after.freelist).toBe(0);
+    expect({ file: statSync(file).size, wal: wal() }).toEqual({ file: after.pages * pageSize, wal: 0 });
+  });
+
   it('remote face: a row written after the call reaches a second connection', async () => {
     const { driver, file } = await withFreePages('remote');
     await driver.reclaimSpace();