Skip to content

fix(driver-turso)!: the remote filter compiler refuses the JSON-column family and answers $contains by membership (#21178) - #21208

Merged
objectstack-fleet[bot] merged 2 commits into
mainfrom
claude/issue-21178-remote-json-column-gate
Oct 1, 2026
Merged

objectstack-fleet[bot] merged 2 commits into
mainfrom
claude/issue-21178-remote-json-column-gate

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Fixes #21178
Clause-②: yes (narrowing)

What changes

RemoteTransport.buildWhereSQL (@objectstack/driver-turso, the filter compiler every remote-mode TursoDriver read and filtered write uses) now applies the JSON-column half of the filter contract exactly as the local face (SqlDriver, which local and replica mode inherit) does, from the same shared home in @objectstack/core:

  • Refusal. On a field the driver stores as a JSON TEXT column, every operator in JSON_COLUMN_INCOMPATIBLE_OPERATORS is refused with INVALID_FILTER / 400 before any statement runs: in an operator map (at the top of the per-operator loop, ahead of every arm), in the bare { field: value } spelling, and in the bare { field: null } spelling. The message and the withheld diagnostic are jsonColumnOperatorRefusalText's output, byte for byte the local face's, through this transport's existing withheld-refusal seam ([A of #7929] a spec-declared provenance mark set at both read-scope merge boundaries, so the driver can restore the author-facing cross-field diagnostic without re-disclosing policy #8220 provenance, diagnostic sink).
  • Membership. $contains / $notContains on such a column answer membership through jsonMembershipPredicate('sqlite', ...) (libSQL is SQLite); the negated form sits inside nullSafeNegative, so a row with no value satisfies $notContains. A scalar string column keeps the substring test.
  • ⛔ No copy of the set, the sentence or the construct in driver-turso; no remote-only dialect.

The widening (for the contract review)

The transport keeps no schema, so it learns "is this a JSON-stored column" the way it learns every other declared fact, through an injected resolver. This adds ONE optional public method on the exported RemoteTransport class, and one type:

  • RemoteTransport.setJsonColumnResolver(resolver: JsonColumnResolver): void
  • type JsonColumnResolver: a function taking (object: string, field: string) and returning boolean, exported from remote-transport.ts and NOT re-exported from index.ts, like its siblings NonTextColumnResolver and DeclaredValueShapeResolver; it reaches the published .d.ts only as the method's parameter type.

TursoDriver's constructor wires it to the inherited SqlDriver.isJsonColumn, beside setDeclaredValueShapeResolver (constructor wiring only in turso-driver.ts; the upsert regions are untouched). In remote mode registerRemoteFieldMetadata calls registerExternalObject, which fills the same jsonFields registry the local face's gate and membership reading ask, so both faces read one population. A RemoteTransport driven standalone without the resolver treats no column as JSON-stored and compiles as before. This is the fourth sibling of setFilterColumnSql, setNonTextColumnResolver and setDeclaredValueShapeResolver; the last of those shipped as "New optional API" in commit fb386074's changeset. The seat's answer to the fork is on the card (claim amendment 5935032674, option A, open to the maintainer's veto).

Why (measured at base 0b12b9ea, on the libsql SQLite stub harness)

Over a multiple: true lookup holding ["u1","u2"] r1, ["u2"] r2, ["u3","u1"] r3, ["u10"] r4 and null r5, the remote face answered, while the local face answered the right-hand column:

filter remote, before local (the contract)
$contains: 'u1' r1, r3, r4 (u10 by substring) r1, r3
$notContains: 'u1' r2, r5 r2, r4, r5
$nin: ['u1'], $ne: 'u1' all five rows (fail-open) INVALID_FILTER / 400
$eq, $in, bare equality no row INVALID_FILTER / 400
$lt / $lte 'u1' r1-r4 (lexicographic over the serialization) INVALID_FILTER / 400
$startsWith: '[', $endsWith: ']' r1-r4 INVALID_FILTER / 400
$icontains: 'u1' r1, r3, r4 INVALID_FILTER / 400
{ owners: null } r5 INVALID_FILTER / 400
json field, $contains: 'u1' r1, r4 (text inside the object) no row (array-only membership)

Pins

  • turso-local-remote-json-column-parity.test.ts (new): one fixture on BOTH faces; every case asserts remote equals local AND the canonical answer. It iterates the refused set from JSON_COLUMN_INCOMPATIBLE_OPERATORS as it stands (27 members at base), asserting code + status + equality with jsonColumnOperatorRefusalText(...).message (never the literal words); the bare infix spellings in an operator map stay refused on both faces. Also: bare equality, null equality ({f: null}, $eq: null, $ne: null), the gate under $and / $or / $not, the population (a tags field and a json field), count(); membership u1 vs ["u10"], u10, $notContains complement with the NULL row, tags, the json object, bind alignment beside sibling predicates, count(); the scalar text-field control ($contains, $nin, $eq, bare and null equality, $startsWith unchanged); $null / $exists still answered.
  • remote-transport-compile-refusal-seam.test.ts: the enumeration requires every seam refusal method to have a row, so jsonColumnOperator gets one per position (operator map, bare value, bare null) on a half-2 transport told exactly one column is JSON-stored. Policy / author / unmarked / merged-arm provenance all hold.

Ablation (one-time proofs, via scripts/ablation-replace.mjs, restore proven by blob hash and empty git diff HEAD)

The suite imports ./turso-driver.js from source (vitest), so no dist/ leg applies.

  • Unwire the resolver in turso-driver.ts (this.remoteTransport.setJsonColumnResolver( replaced by a no-op call; anchor 1 to 0, blob a9affc72 to b90200ef): the parity suite goes 24 failed / 19 passed of 43, as predicted. Red: all 14 operator-map refusals, bare equality, depth, population, both count() pins, u1 membership, $notContains, the json object, bind alignment, null equality. Still green, as predicted: the 13 bare-infix rows (both faces refuse as an object comparand whatever the gate), the set check, u10 and tags membership (substring coincides), the scalar controls, presence.
  • Disable the membership arm in remote-transport.ts (pushJsonMembership answers false): 5 failed / 38 passed, exactly the five membership pins whose substring answer differs; every refusal pin stays green.
  • Restored: git diff HEAD empty, blob equals HEAD in both legs.

Filter-semantics compile surfaces, one conclusion per face

  1. driver-sql applyFilterCondition (sql-driver.ts): already conformant — it is the contract; the parity suite's local column is its answer, and driver-sqlite-wasm and local/replica driver-turso inherit it. Not edited.
  2. driver-turso RemoteTransport.buildWhereSQL (remote-transport.ts): changed (this PR).
  3. service-analytics compileScopedFilterToSql (read-scope-sql.ts): out of scope — another face; it already reaches the shared membership construct through contains-membership-sql.ts (imports jsonMembershipPredicate).
  4. service-analytics lowerAnalyticsWhere (strategies/filter-normalizer.ts): out of scope — another face, same shared construct as 3.
  5. formula matchesFilterCondition (matches-filter.ts): out of scope — another face, and seat 2's in-flight #5930 step 4 (domain:engine): the engine-fed faces delete their hand-copied filter meaning (driver-sql, turso remote, memory query, mongodb, formula, having); the memory reference matcher retires (D6) #20822 group 3b owns it.
  6. objectql applyHaving / matchesHaving (having-filter.ts): out of scope — another face (it already imports the shared set and sentence), and seat 2's #5930 step 4 (domain:engine): the engine-fed faces delete their hand-copied filter meaning (driver-sql, turso remote, memory query, mongodb, formula, having); the memory reference matcher retires (D6) #20822 group 3b owns it.
  7. driver-memory query path (refuses the family since 45ce12a4, filter-refusal.ts imports the shared set) and driver-mongodb translateFieldOperators (mongodb-filter.ts): out of scope — other faces, not touched.

Tests and gates (all on HEAD c67a136e unless noted)

  • pnpm --filter @objectstack/driver-turso exec vitest run --maxWorkers=2: 84 files passed, 2286 tests passed, 33 skipped (exit 0, via os-verify-lock).
  • pnpm --filter @objectstack/driver-turso typecheck: exit 0; tsc --listFiles includes both edited test files.
  • pnpm check:driver-conformance: before (base 0b12b9ea) 50 covered / 0 DEBT / 0 exempt; after (c67a136e) 50 covered / 0 DEBT / 0 exempt. The ledger did not move.
  • node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands derived 63 commands at c67a136e; all 63 were run and reconciled with --ran (each line recording its exit code): 62 run, 1 NOT MEASURED. pnpm check:dual-build-cjs-loads exited 3 (PREREQUISITE NOT MET: it reads every package's dist/, and a repo-wide build is CI's); narrowed in its place, require of driver-turso's CJS build and import of its ESM build both load and expose setJsonColumnResolver. check-plugin-teardown-shape --self-test and check:lean-entry-closure first exited 3 on prerequisites (a fixture commit outside the shallow clone; objectql's dist/) and exited 0 after fetching that commit and building objectql's closure.
  • check-adr-0087-registration --base origin/main: the changeset reads as declared-breaking with one disposition, not-required (no-migration-prescription); check-changeset-no-major: no major bump.
  • Lint, narrowed to the 4 touched TS files: eslint --no-inline-config --format json read 4 files, 0 errors, 0 warnings. All four are in eslint .'s population (--print-config resolves each). The config never enables type-aware linting (the resolved parserOptions are ecmaVersion and sourceType only, and eslint.config.mjs says so in its header), so this diff cannot move a verdict on any untouched file. The full pnpm lint is CI's.
  • Downstream consumers (cli, qa/dogfood, runtime, service-datasource) are declared to CI: none references RemoteTransport's API, and the one remote-mode consumer test (date-bucket-parity-turso) aggregates without a JSON-field filter.

Changeset

.changeset/21178-remote-json-column-gate.md: @objectstack/driver-turso minor, BREAKING banner, Clause-②: yes (narrowing), ADR-0087 disposition not-required (no-migration-prescription), the new optional API named, and the migration text (write $contains for "holds this member", an $or of $contains for any-of, $not around either for the exclusion, $null / $exists / $empty for presence).

Siblings

Acceptance notes

  • The local face's gate reads the operator, not the comparand, so on a JSON column it refuses the equality spellings of a null comparand ({ f: null }, $eq: null, $ne: null) although IS NULL is a well-formed presence question there; this PR matches it on the remote face (that is the parity invariant), and $null / $exists / $empty remain the presence spellings on both. Noted, not filed: no declared contract says otherwise ($eq is a declared member of the set). Carrier: none.
  • Refusal order differs from the local face only for a doubly-wrong filter: the remote gate runs ahead of each arm's own comparand gate (the remote comparand gate lives inside each arm), so { owners: { $in: [{ ... }] } } reads the JSON-column sentence remotely and the comparand sentence locally. Both are INVALID_FILTER / 400.
  • The bare infix spellings (=, in, …) in an operator map are refused on both faces as an object comparand, with different wording per face; pre-existing, not JSON-specific.

Generated by Claude Code

claude added 2 commits October 1, 2026 15:53
… gate and answers $contains by membership

RemoteTransport.buildWhereSQL now refuses every operator in @objectstack/core's
JSON_COLUMN_INCOMPATIBLE_OPERATORS on a column the driver stores as JSON text,
with the shared jsonColumnOperatorRefusalText sentence, and answers
$contains / $notContains there through jsonMembershipPredicate('sqlite'), the
negated form NULL-safe. The JSON-column population is the driver's own
jsonFields registry, handed down through a new optional
RemoteTransport.setJsonColumnResolver that TursoDriver wires to the inherited
SqlDriver.isJsonColumn, beside setDeclaredValueShapeResolver.

A local/remote parity suite holds both faces to one answer over the whole
shared set, the membership pair, the bare and null equality spellings and a
scalar text-field control.

Claude-Session: https://claude.ai/code/session_017xfMoEjKUuSh2xYB8sCozp
Co-authored-by: Claude <noreply@anthropic.com>
…-refusal seam table; add the changeset

The seam enumeration requires every refusal method that goes through the
withheld seam to have a row: jsonColumnOperator gets one per position (operator
map, bare value, bare null), on a half-2 transport told that exactly one column
is JSON-stored. The class is read from jsonColumnOperatorRefusalText, never
spelled in the test. The bare-null position of buildWhereSQL now asks the gate
too, as the local face's bare-value positions do.

Claude-Session: https://claude.ai/code/session_017xfMoEjKUuSh2xYB8sCozp
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions github-actions Bot added the size/l label Oct 1, 2026
@github-actions github-actions Bot added documentation Improvements or additions to documentation tests tooling labels Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

8 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see
  • 1 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 54 of 206 client-bound route-ledger rows — the other 152 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 152: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 97 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
  • a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 7 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 0d421041d14cfc716e85e9bdd1da3d57d4eb3aa1 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 7213aa27b0367f41d157a772dad1e41ddc8a7626 — the merge of head c67a136e94b876497ea177a6b49cd12fd3b3eefe into base 0d421041d14cfc716e85e9bdd1da3d57d4eb3aa1, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 7213aa27b0367f41d157a772dad1e41ddc8a7626 && git checkout 7213aa27b0367f41d157a772dad1e41ddc8a7626
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 0d421041d14cfc716e85e9bdd1da3d57d4eb3aa1 c67a136e94b876497ea177a6b49cd12fd3b3eefe && git checkout -B drift-repro 0d421041d14cfc716e85e9bdd1da3d57d4eb3aa1 && git merge --no-ff c67a136e94b876497ea177a6b49cd12fd3b3eefe

node scripts/docs-audit/affected-docs.mjs --json 0d421041d14cfc716e85e9bdd1da3d57d4eb3aa1

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

Contract review

Served-tier: CONTRACT_REVIEW_TIER
Head-sha: c67a136e94b876497ea177a6b49cd12fd3b3eefe
Local-runs: none

Inputs read: card #21178 (body and all five comments: triage 5933993910, claim 5934703166, os-dev-report 5934971162, claim amendment 5935032674, os-dev-report 5935977677); PR #21208 body, file list (5 files, +553/-0) and the net diff; the check-runs on the head. Verified against origin/main at 5e5ce48c by git show / git grep reads only: the three-dot diff origin/main...c67a136e equals the PR's five-file list, and no commit on main after the branch base 0b12b9ea touches packages/drivers/driver-turso, the two @objectstack/core modules the diff imports, or driver-sql's sql-driver.ts.

① Derived judgments

Accept-set changes, all on the REMOTE face of TursoDriver (RemoteTransport.buildWhereSQL, the one where compiler every remote-mode door shares):

  1. Operator-map gate — on a JSON-stored column every member of JSON_COLUMN_INCOMPATIBLE_OPERATORS (27 at base) is refused INVALID_FILTER / 400 at the top of the per-operator loop, ahead of every arm. Right. Same position as SqlDriver.assertOperatorAppliesToColumn on the local face (the operator-map branch of sql-driver.ts), the set read from @objectstack/core, no copy in driver-turso (the diff's only new imports are the three shared symbols). One ordering difference, named in the PR's acceptance notes: the local face runs its comparand gate before the column gate, the remote face runs the column gate before each arm's comparand gate, so a filter wrong in both ways reads the JSON sentence remotely and the comparand sentence locally. Both are the same ADR-0112 envelope through the same withheld seam; accepted, not a fork of the accept set.
  2. Bare { field: value } gate — implicit = refused on a JSON column, after the comparand gate. Right. The local face's bare-value loop does the same with bare = true, in that order.
  3. Bare { field: null } gate — refused on a JSON column instead of compiling IS NULL. Right. The local face's top-level loop does not branch on null; its = gate fires whatever the comparand, so the previous remote answer (the NULL row) was the fork the card is about. $null / $exists / $empty stay the presence spellings on both faces and the parity suite pins them (§4). The changeset says "whatever the comparand, null included", so a consumer is told.
  4. $contains on a JSON column — membership through jsonMembershipPredicate('sqlite', ...). Right. libSQL is SQLite; the spec's contract (filter.zod.ts, "A JSON-stored column changes what $contains ASKS") declares membership on a multiple: true field or a JSON_COLUMN_TYPES member and substring on a scalar column; the column is emitted as the plain quoted identifier (no storage-form rewrite, as locally), the comparand is handed over raw (as applyJsonMembership does), binds are pushed in placeholder order; the arm order (comparand gate, non-text gate, membership, substring) is the local face's order.
  5. $notContains on a JSON column — NOT of the same construct inside nullSafeNegative. Right. The exact twin of applyJsonMembership with negate = true through applyNullSafeNegative; the complement includes the NULL row, pinned.
  6. The missing-construct guard — a thrown error rather than a fall-through to the substring emitter when the predicate returns no construct. Right (loud, not a fallback); unreachable for the constant 'sqlite' dialect.
  7. Scalar columns, and a transport handed no resolver — unchanged. Right. isJsonColumn answers false without a resolver; the parity suite's scalar title control and the seam test's other rows hold the old answers.
  8. The population — the driver's own jsonFields registry via the inherited SqlDriver.isJsonColumn, filled in remote mode by registerRemoteFieldMetadata calling registerExternalObject (keyed by object name). Right, and it is what triage's "no third copy" and the card's "no remote-only copy" required; no re-derivation from field types in the transport.
  9. Doors — the gate sits in the one compiler, so find, findOne, count, updateMany, deleteMany, aggregate and distinct are covered as the changeset says; the parity suite pins find and count. Right.

Public-surface changes:

  1. RemoteTransport.setJsonColumnResolver(resolver) — one new optional public method on a class index.ts exports. Right to book as a widening (Clause-②: yes): the precedent is commit fb386074 (filter: the engine's compile surfaces answer $empty by the field's declared type (driver-sql and heirs, turso remote, driver-memory, driver-mongodb, formula, objectql having) — ruling A on #20399 #20444), which shipped setDeclaredValueShapeResolver as "New optional API" under Clause-②: yes (widening). driver-turso carries no api-surface baseline, so no gate records this; this record is the control. The wiring in turso-driver.ts is 11 lines in the constructor directly after setDeclaredValueShapeResolver, and the upsert regions are untouched ([security] driver upsert: a tenant-scoped upsert keyed on a globally-unique business column can merge into, and re-parent, another tenant's row #21185 stays region-disjoint).
  2. type JsonColumnResolver — exported from remote-transport.ts, not re-exported from index.ts (index.ts is not in the file list), so it reaches the published .d.ts only as the method's parameter type, exactly as NonTextColumnResolver and DeclaredValueShapeResolver do (index.ts re-exports only FilterColumnSqlResolver). Right, and the PR body states it accurately.
  3. Nothing else moves: no edit to @objectstack/core, driver-sql, spec, docs or workflows; none of the five paths is in GOVERNED_SURFACES; no hand-written page enumerates the transport's setters or states a remote-mode filter semantic, so no docs drift is owed. Right.

Pins:

  1. turso-local-remote-json-column-parity.test.ts iterates the refused set from the module (with a non-emptiness guard so the loop cannot be vacuous), asserts remote equals local AND the canonical answer, compares messages by equality with jsonColumnOperatorRefusalText(...) so [finding] driver-sql's JSON-column refusal is about 800 characters and is cut at the REST envelope's 500, so no caller reads the sentence saying the field and operator were withheld #21067's reword cannot flip them, and covers the card's table, u1 against ["u10"], the scalar control, the tags and json population, depth under $and / $or / $not, count(), bind alignment and presence. Right.
  2. remote-transport-compile-refusal-seam.test.ts: jsonColumnOperator goes through withheldRefusal, so the enumeration half requires a DOORS row; the diff adds one DOORS row and two ARMS rows (bare value, bare null) and injects a resolver naming exactly one column, so every other row compiles as before. Right.

Check-runs on c67a136e, read 2026-10-01T16:50:27Z: Build Core success; Dogfood Regression Gate success (shards 1/3, 2/3, 3/3 success); Temporal Conformance (live PG + MySQL) success; Governed Surface Queue Guard success; Test Core shards 2/6 through 6/6 success, 1/6 in progress; Type Check · consumer gates, Type Check · debt ledger, Type Check · source gates success, Type Check · workspace in progress, and the TypeScript Type Check aggregator they feed not yet created on this head; Lint & Repo Gates in progress; Check Changeset success (the job that runs check-adr-0087-registration and check-changeset-no-major against the merge base); Check PR Size, Check Documentation Links, the three card-claim guards, Auto Label, Dogfood Verify CLI and filter success; Build Docs, Console Pin Gate and Packed-tarball smoke skipped. No run failed or was cancelled. The in-progress required contexts are not yet verdicts; the queue enforces them, and that is not a FAIL here.

② Semver level

.changeset/21178-remote-json-column-gate.md: @objectstack/driver-turso minor, a BREAKING banner, the migration text in prose, and exactly one ADR-0087 marker reading not-required (no-migration-prescription). Matches what the diff publishes.

Clause-②: yes (narrowing)

Read as scripts/pm/clause2-line.mjs reads it: yes because setJsonColumnResolver widens @objectstack/driver-turso's exported surface; (narrowing) because the remote face's accept set narrows (27 operators refused on a JSON-stored column) and $contains / $notContains change rows there. The PR body and the changeset carry the same line, and it matches the claim amendment 5935032674.

③ Boundary flags

Implemented-by: claude/issue-21178-remote-json-column-gate
Reviewed-by: session_017xfMoEjKUuSh2xYB8sCozp

VERDICT: PASS


Generated by Claude Code

@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review October 1, 2026 16:57
@objectstack-fleet
objectstack-fleet Bot enabled auto-merge October 1, 2026 16:58
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Oct 1, 2026
Merged via the queue into main with commit 862f12c Oct 1, 2026
36 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/issue-21178-remote-json-column-gate branch October 1, 2026 17:18
akarma-synetal pushed a commit to akarma-synetal/framework that referenced this pull request Oct 7, 2026
…es a REST caller whole (objectstack-ai#21213)

Fixes objectstack-ai#21067

Clause-②: no

## What this changes

`jsonColumnOperatorRefusalText`
(`packages/core/src/utils/json-column-operator-refusal.ts`) is the one
builder of the `INVALID_FILTER` / 400 refusal a filter gets for a scalar
comparison or text operator on a multi-value or JSON field. It is
rewritten once, for both of the card's reasons together (seat 2's
carried note folded in: one rewrite, not two passes):

1. **Under the REST bound.** The withheld message was 748 characters.
The `/data` door cuts a 4xx message of 500 or more to 499 plus an
ellipsis, so the wire ended `…Refused rather than compiled because the
answ…`, and no caller read the sentence saying the field and the
operator were withheld. It is now **486** characters, one constant text.
2. **True on every face.** The old reason named `driver-sql`'s storage
("a field this driver stores as a JSON TEXT column") and the two wrong
answers SQL used to give. Both are untrue on the engine's
per-aggregation `filter` and on `driver-memory`, which print the same
text. The reason is now the field's declaration: `it aims a scalar
comparison or text operator at a multi-value or JSON field, which it
cannot test for one member.`
3. **The null comparand's repair (added at the PM's note on this
card).** The refused set also catches `{ f: null }`, `$eq: null` and
`$ne: null`, which ask whether the field has a value; `$contains` cannot
express that. One constant clause, with no branch on the comparand,
names the presence spellings: `For no value, use "$null" or "$empty".`
Both answer on a multi-value or JSON field on every face (they are
outside the refused set), and the REST pin now proves `$null: true`,
`$null: false` and `$empty: true` answer on both the `where` and the
per-aggregation faces.

The new message, in full:

```text
A constraint in this filter WAS NOT APPLIED: it aims a scalar comparison or text operator at a multi-value or JSON field, which it cannot test for one member. Use "$contains" for membership ({ "FIELD": { "$contains": "a" } }), or an $or of "$contains" for any-of ({ "$or": [{ "FIELD": { "$contains": "a" } }, { "FIELD": { "$contains": "b" } }] }). For no value, use "$null" or "$empty". The field and the operator are withheld from the message; the full diagnostic is in the server log.
```

The diagnostic (server log, and the author-disclosed wire text, see
below) shares that reason, names the operator in it (`it aims "$in", a
scalar comparison or text operator, at …`; the bare spelling's operator
as `=`, as before), names the field, and spells the remedy with the
field's name, presence clause included. It drops the same storage and
SQL history. One `refusalReason()` and one `containsRemedy()` (which
ends in `PRESENCE_REMEDY`) serve both texts, so the two cannot drift.
The SQL mechanism and the measured wrong answers stay in the builder's
docblock.

**Unchanged:** `code`, `status`, the refused operator set
(`JSON_COLUMN_INCOMPATIBLE_OPERATORS`, 27 spellings), the `$contains`
remedy, no new export, no new code.

**Faces.** Re-derived at `cb45469e` by name and by text: every face that
prints this sentence calls the builder, and none builds its own copy.
They are `driver-sql` `jsonColumnOperatorError` (`sql-driver.ts`),
`objectql` `having-filter.ts` (the judgment and the per-row backstop,
three call sites), and `driver-memory` `jsonStoredFieldOperatorError`
(`filter-refusal.ts`). No face file changes except one stale docblock in
`sql-driver.ts`. `driver-turso`'s `remote-transport.ts` and
`driver-mongodb` share phrases with other refusals, not this sentence.
No docs page or skill quotes it (`content/docs/**`, `skills/**`,
`docs/**`, `apps/docs/**`, by builder name and by seven distinctive
phrases: zero hits).

## Measured, before and after

| reading | `BASE` `cb45469e` | this PR |
|:--|:--|:--|
| withheld message, constant | 748 chars | 486 chars |
| its wire `error`, SQLite and live PostgreSQL 16.14, `mapDataError` |
499 plus an ellipsis, ending `Refused rather than compiled because the
answ…` | the whole message |
| author-disclosed diagnostic, field `owners`, `$in` | 643 chars | 402
chars |
| that, at `POST /api/v1/data/:object/query` through the real
`SecurityPlugin` | 499 plus an ellipsis, ending `Refused rather than
compiled beca…` | the whole diagnostic (measured at 377 before the
presence clause; the 402 text is pinned whole through `mapDataError`) |
| field-name length from which the diagnostic is cut on the wire | every
length | 27 characters (the presence clause goes first) |
| field-name length from which the cut takes the any-of example | 10
characters | 37 characters |

### A dispatch hypothesis this falsified (H5)

The dispatch read the diagnostic as server-log text the envelope never
bounds. Measured, it is also a wire text. `driver-sql`'s `'author'`
provenance arm (`resolveWithheldFilterRefusal`) swaps the refusal for
the diagnostic when the predicate is the caller's own, and
`plugin-security` marks the caller's verbatim `where` `'author'`
(`security-plugin.ts`, the `markFilterSubtreeProvenance(callerWhere,
'author')` call). Through `POST /api/v1/data/:object/query` with a real
`SecurityPlugin`, a member caller sending `{ owners: { $in: ['u1'] } }`
received the diagnostic: cut to 500 at `BASE`, whole (377) at
`caf0e3c7`, before the presence clause raised it to 402. That was a
one-off measurement file, run twice and not committed; the `BASE` leg
rebuilt `core`'s `dist/` with the old text and restored it, with dist
preflight proofs both ways.

The diagnostic was in the rewrite already, because its reason clause was
the same untrue `driver-sql` mechanism seat 2's note names. All four
conditions of the bounded in-place fix hold: same defect class (a
refusal cut at the envelope), a mechanical fix in the shape triage
pinned, the claimed file, and the same gate families. So it is fixed
here, not filed. Its content is kept: field and operator named, remedy
with the field's name.

## Pins

New, compared with the builder's output and the bound's own function
(`truncateClientMessage` / `mapDataError` from `@objectstack/types`),
never with a copied sentence or a retyped 500. `CLIENT_MESSAGE_MAX`
itself is module-private in
`packages/types/src/data-error-classification.ts`, so it is not
imported; `truncateClientMessage` is already exported there for `rest`,
and nothing new is exported. A sibling that rewords nothing and only
calls the builder cannot flip these.

- `packages/core/src/utils/json-column-operator-refusal.test.ts`:
re-captured hashes and lengths. For every refused spelling,
`truncateClientMessage(message)` returns it unchanged. The message
carries the one-member, any-of and no-value remedy and ends with the
withheld sentence. Neither text names a storage form or a backend's
wrong answer. The diagnostic gives the same reason with the operator
named.
-
`packages/drivers/driver-sql/src/sql-driver-json-column-refusal-wire-bound.test.ts`
(new, `DIALECT_CELLS`): for 14 operators plus bare equality, on a
multi-value lookup, a `tags` field and a `json` field,
`mapDataError(err).body.error` equals the shared message (unmarked,
carrying the presence clause) and the shared diagnostic (author-marked),
with the remedy spelling. SQLite always; **PostgreSQL and MySQL in CI's
`Temporal Conformance (live PG + MySQL)` job**, which runs this
package's whole suite with both URLs set.
- `packages/rest/src/aggregation-filter-json-column-refusal.test.ts`:
through `POST /api/v1/data/:object/query`, the `where` twin's body
equals the builder's message whole, not just the per-aggregation face's
(`toBe`, plus the any-of remedy, the presence clause and the withheld
sentence), on the null-comparand rows too; three new controls show
`$null: true`, `$null: false` and `$empty: true` answer with equal
counts on both faces.

Flipped, each to the new substance:

- `sql-driver-json-column-operator-refusal.test.ts`: `'JSON TEXT
column'` becomes the operator-named reason, and the remedy is asserted
with the field's own name.
- `sql-driver-target-field-provenance.test.ts`: the class fragment that
survives redaction, `'JSON TEXT column'`, becomes `'at a multi-value or
JSON field'`.
- `sql-driver-json-column-refusal-shared-text.test.ts`: the docblock's
"did not change by one byte" claim.

ADR-0112 `code` plus `status` assertions are untouched everywhere.

### Reverse verification

At `fefb6e1f`, `BASE`'s builder was written to disk (tree only, never
staged) and its landing checked by grep: old text 1, new 0. Then:

- `sql-driver-json-column-refusal-wire-bound.test.ts` (SQLite plus live
PostgreSQL 16.14): **90 failed** (45 per cell), 1 skipped (MySQL). The
red direction, as expected.
- The core pin: **8 failed**, 4 passed.

Restored with `git checkout HEAD -- PATH` under an `EXIT INT TERM` trap.
The blob is `2f17d7f3`, equal to HEAD's, and `git diff HEAD` is empty.

## Verification (on `d352319a`, the final head: the branch, one merge of
`origin/main` `0d421041`, which touches none of these packages, and the
presence-clause commit)

The targeted files below ran on `d352319a`, with `OS_TEST_POSTGRES_URL`
set to a private PostgreSQL 16.14:

- core: 12 passed
- driver-sql, the five JSON-column files: 360 passed, 2 skipped (the
MySQL cells)
- rest `aggregation-filter-json-column-refusal`: 136 passed, 68 skipped
(MySQL)
- driver-memory `memory-20444-*` and `memory-21066-*`: 125 passed
- objectql `engine-aggregate-filter-json-column-refusal`,
`engine-aggregate-filter`, `engine-aggregate-filter-array-membership`
and `engine-cascade-delete-multivalue-probe`: 263 passed

Full suites, on the pre-merge commits (before the presence clause; that
commit touches only the builder, its unit pin and the two wire pins
above, which were re-run):

- `@objectstack/core` local: 74 files, 2107 passed
- `@objectstack/driver-sql`, SQLite: 208 files passed, 11 skipped; 3469
passed, 192 skipped
- `@objectstack/rest` local: 255 files, 4834 passed, 301 skipped

Typecheck: `core`, `driver-sql` and `rest` all pass, and `tsc
--listFiles` confirms the four touched driver-sql test files are in its
program.

**Gates** (`node scripts/pm/dispatch-gates.mjs --repo
objectstack-ai/objectstack --commands`, re-derived and re-run on
`d352319a`): 65 derived, 65 exit 0, 0 NOT MEASURED. `--ran` with exit
codes reconciles to 65 accounted, 0 unrun.

- `check:dual-build-cjs-loads` measured this time: 105 require entries
across 66 packages load. The first run on `8450f66b` was exit 3
`PREREQUISITE NOT MET`, since a whole-repo build was missing then.
- `check:driver-conformance`, before the first edit and after the last
commit: 50 covered, 0 DEBT, 0 exempt; dialect axis 8 suites, 0 in the
DIALECT ledger. The ledger did not move.

**Lint**, narrowed:

1. The population comes from eslint's own config: `--format json`
reports every listed file with 0 warnings, so none was ignored.
2. On `d352319a`, the 8 `.ts` files of this diff were linted, with 0
errors and 0 warnings.
3. `eslint.config.mjs` enables no type-aware linting (no
`parserOptions.project`), so this diff cannot move the verdict on any
untouched file.

The repo-wide `pnpm lint` is CI's to run.

**Patch round 1 (`89209290`).** `Lint & Repo Gates` was red at
`d352319a` on `pnpm check:live-db-isolation`:
`packages/rest/src/aggregation-filter-json-column-refusal.test.ts:258
use names the literal "$null"`. The fix is test-only: the presence
clause is asserted in two pieces. On `89209290`:

- `pnpm check:live-db-isolation`: exit 0, 34 live-server files scanned.
- That test file: 136 passed, 68 skipped, with SQLite and live
PostgreSQL 16.14 running and MySQL a named skip.
- The 65 derived families re-run: 65 exit 0.
`check:dual-build-cjs-loads` gave exit 3 PREREQUISITE NOT MET until
another family built the missing `dist/`, then exit 0 on re-run. `--ran`
reconciles to 65 run, 0 NOT MEASURED.
- `check:live-db-isolation` is a declared WIDE-population family, so
per-card derivation never names it.

## Siblings that print this sentence (both landed before this PR; seat
edit)

Both siblings this PR named as later landers merged first, so they are
earlier landers now:

- objectstack-ai#21178 → `862f12c0b` (PR objectstack-ai#21208): the `driver-turso` remote face. It
prints the builder's output and pins it by equality with the builder, so
this rewrite does not flip it.
- objectstack-ai#20822 group 3b → `e18fea6dc` (PR objectstack-ai#21196): new `having-filter.ts`
callers of the builder.

This branch is behind both. The merge queue rebuilds it onto `main`, so
their pins run against this text in the merge group.

## Acceptance notes

- **The diagnostic's boundary.** It names the field four times, so its
length grows with the name. It is whole on the wire up to 26-character
field names (measured over `$in`, `$startsWith` and bare equality). From
27 characters the cut takes the presence clause first, and from 37 the
any-of example; the one-member remedy and both names come before both.
Field names declare no maximum length, so no text that repeats the name
can be bounded. The withheld message, the card's subject, is constant
and bounded.
- **A gate false positive, on the test's spelling.**
`check:live-db-isolation`'s statement-head needle matches the verb USE
followed by a quoted operand (`STATEMENT` in
`scripts/check-live-db-isolation.mjs`). It read the assertion string
`'For no value, use "$null" or "$empty".'` at
`packages/rest/src/aggregation-filter-json-column-refusal.test.ts:258`
as a MySQL USE naming a database `"$null"`. Fixed in `89209290` by
asserting the clause in two pieces; the gate and the product sentence
are unchanged.
- `packages/objectql/src/engine-cascade-delete-multivalue-probe.test.ts`
keeps a test double whose refusal paraphrases the old wording ("is
stored as a JSON TEXT column"). It asserts only `code` and `status`, so
it is not a pin of this text. Left as is.
- `rest`'s own PostgreSQL and MySQL cells of
`aggregation-filter-json-column-refusal.test.ts` are still provisioned
by no CI job (its header says so). The PostgreSQL wire pin that CI does
run is the new `driver-sql` file.
- The local PostgreSQL leg ran against a private PostgreSQL 16.14
started for this run on a random port (UTC server, so `driver-sql`'s
temporal files were not run against it). It was stopped afterwards.

---
_Generated by [Claude
Code](https://claude.ai/code/session_017xfMoEjKUuSh2xYB8sCozp)_

---------

Co-authored-by: Claude <noreply@anthropic.com>
akarma-synetal pushed a commit to akarma-synetal/framework that referenced this pull request Oct 7, 2026
…its counter from the stored MAX on SQLite (objectstack-ai#21163) (objectstack-ai#21206)

Fixes objectstack-ai#21163

Clause-②: no

## What changed

`SqlDriver.scanMaxNumericTail`
(`packages/drivers/driver-sql/src/sql-driver.ts`) is the one read of the
highest counter already stored under an autonumber prefix. Its two
callers are the cold bootstrap in `getNextSequenceValue` and the objectstack-ai#5495
re-seed `resyncSequenceToDataMax`. It escapes the rendered prefix with
`escapeLikePrefix` (a backslash before `\`, `%` and `_`). It then
compiled `where(field, 'like', pattern)` through Knex, which declares no
`ESCAPE` clause on any dialect.

The scan now reads:

```ts
.whereRaw('?? like ? escape ?', [field, `${this.escapeLikePrefix(prefix)}%`, LIKE_ESCAPE_CHARACTER])
```

`LIKE_ESCAPE_CHARACTER` is the driver's existing bound escape character,
the one the filter compiler's `LIKE ... ESCAPE ?` already binds. It is
bound and never written as a literal, because MySQL applies C escape
syntax inside string literals. There is one escape helper
(`escapeLikePrefix`, unchanged) and one declared escape character, the
same on every dialect. The method's docblock records why. One file of
production code changed. The Turso remote face's own statement
(`turso-driver.ts`, which already declares `ESCAPE` with a backslash) is
untouched.

## Why

SQLite's `LIKE` has no escape character unless one is declared. Postgres
and MySQL use a backslash by default. So on the SQLite faces, a prefix
carrying `_`, `%` or `\` matched no stored row. Measured at `cb45469e`
with this PR's pins and the pre-fix builder restored:

| format | stored before the create | cold create issued | create after
rows 2..30 land above a warm counter |
|:---|:---|:---|:---|
| `SO_{0000}` | `SO_0007` | `SO_0001` | refused: `UNIQUE constraint
failed` (on `SO_0005`, retries spent) |
| `SO%{0000}` | `SO%0007` | `SO%0001` | refused, same |
| `SO\{0000}` | `SO\0007` | `SO\0001` | refused, same |
| `{region}-{0000}`, region `north_east` | `north_east-0007` |
`north_east-0001` | not pinned |
| `SO-{0000}` (control) | `SO-0007` | `SO-0008` | `SO-0031` |

The `{region}` row shows that the trigger does not need an unusual
format. The prefix is rendered, so a `_` can come from a field value
such as a snake_case machine name.

## Pins

-
`packages/drivers/driver-sql/src/sql-driver-21163-autonumber-prefix-like-escape.test.ts`:
one suite per dialect cell through `declareDialectCell`. SQLite always
runs. Live Postgres and MySQL run where provisioned (`Temporal
Conformance (live PG + MySQL)` runs this whole package with
`OS_EXPECT_LIVE_DIALECT_MATRIX=1`) and are declared un-run otherwise.
Each cell covers `_`, `%` and `\` prefixes plus a plain control, each
cold and on the objectstack-ai#5495 re-seed, plus the `{field}`-borne `_`.
-
`packages/drivers/driver-turso/src/turso-autonumber-prefix-like-escape.test.ts`:
the LOCAL face (`url: ':memory:'`) and the REMOTE face (over
`makeLibsqlSqliteStub`, a real SQLite behind the libsql client), each
with `_`, `%` and a plain control, cold and on re-seed. The REMOTE legs
passed before this fix because that statement already declared its
escape. They hold both faces to one answer from the one helper.

The assertion is the issued autonumber as stored, read back through the
driver's `findOne` by an id the test chose. On MySQL `create` returns
the insert id rather than the row, so its return value is not a reading
every cell can make. Never a SQL string.

## Reverse verification (fix committed first, at `9b3f7d0a`)

Mutation: `node scripts/ablation-replace.mjs` replaced the new
`whereRaw(...)` with the pre-fix `where(field, 'like', ...)` (anchor 1
to 0, blob `cb63656d` to `5f8edc7a`). Restore: blob equal to HEAD, `git
diff HEAD` empty, whole-tree `git status --porcelain` clean.

- driver-sql pin over src: **7 failed / 2 passed**. All three escaped
prefixes failed cold and on re-seed, and so did the `{field}` case
(`expected 'SO_0001' to be 'SO_0008'`, `UNIQUE constraint failed`). The
two controls passed.
- driver-sql rebuilt, and `node scripts/ablation-dist-preflight.mjs
@objectstack/driver-sql '?? like ? escape ?' --absent` confirmed the fix
was absent from all 6 built files. Then the turso pin: **4 failed / 8
passed**. LOCAL `_` and `%` failed cold and on re-seed. LOCAL control
and all 6 REMOTE legs passed.
- Restore leg: rebuilt, and preflight `'where(field, "like"' --absent`
confirmed the mutation was gone from dist (tree clean). Pins: driver-sql
9 passed / 2 skipped (live cells unprovisioned), turso 12 passed. The
pristine dist reading (default mode) found `?? like ? escape ?` present
in `dist/index.js` and `dist/index.mjs`.

### Live Postgres (local PG 16, `initdb` + `pg_ctl` on a scratch port,
torn down after)

- With the fix: 18 passed / 1 skipped (9 sqlite + 9 live postgres; mysql
un-run).
- With the pre-fix builder restored: 7 failed, all in the sqlite cell. 0
failed in live postgres.

So declaring the escape does not change behavior on Postgres, which
already used a backslash by default. **MySQL: NOT MEASURED locally**
because this container has no MySQL server. The same file's live mysql
cell runs in `Temporal Conformance (live PG + MySQL)`.

## Verification (at HEAD `7b7fcafb`)

- **Patch round 1 (`5b83098b`, seat edit):** the live MySQL cell failed
on `create` return values (insert id `0` on MySQL). The pin now reads
the stored row. The whole `driver-sql` suite, CI-shaped against a local
live PostgreSQL 16 and MySQL 8.0.46: 219 files, 5108 passed / 1 skipped.
Reverse verification on the reworked pin: 7 failed, all in the SQLite
cell, 0 in live PostgreSQL and live MySQL.

- `pnpm --filter @objectstack/driver-sql test`: exit 0, 208 files passed
/ 11 skipped, 3433 tests passed / 192 skipped.
- `pnpm --filter @objectstack/driver-turso test`: exit 0, 84 files
passed, 2243 tests passed / 33 skipped.
- `typecheck` on both packages: exit 0. `tsc --listFilesOnly` includes
both new test files in each package's program.
- `pnpm check:driver-conformance`: identical before the first edit (at
`cb45469e`) and after the last commit: 50 covered cells, 0 in the DEBT
ledger, 0 exempt. Dialect axis: 8 suites, 0 in the DIALECT ledger.
- `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack
--commands` (no paths) derived 63 commands. 61 exited 0.
`check:type-check-debt` hit a 280 s per-command cap on the first run and
exited 0 on a re-run (re-measure OK in 211 s, self-test OK). Two exited
3 (PREREQUISITE NOT MET, NOT MEASURED): `check:dual-build-cjs-loads`
needs a whole-repo build, and `check:lean-entry-closure` needs
objectql's dist, which is outside this diff's closure. For the first
one, both changed packages' CJS builds were loaded directly with
`require` and loaded OK. `--ran` reconciliation: 63 derived, 61 run, 2
NOT-MEASURED, 0 UNRUN.
- Lint was narrowed to the 3 changed TS files: `eslint
--no-inline-config --format json` gave 3 files, 0 errors, 0 warnings.
Population: `eslint.config.mjs` lints `**/*.{ts,...}` outside its
never-linted build dirs, and no file reported as ignored. Invariance:
the config enables no type-aware linting (no `parserOptions.project`),
so this diff cannot move the verdict on any untouched file. Repo-wide
`pnpm lint` is CI's.

## Acceptance notes

- `escapeLikeComparand` (the filter compiler's module function) and
`escapeLikePrefix` (this method) escape the identical set: a backslash
before each of `\`, `%` and `_`. They were not merged. Triage named
`escapeLikePrefix` as this path's one helper, and nothing here needed a
merge.
- The Turso remote face writes its escape as an SQL literal. That is
correct, because it only ever talks to SQLite. Its new REMOTE legs are
the guard that keeps the two faces' escape characters equal.
- `SqliteWasmDriver` extends `SqlDriver` and inherits the scan. A
one-off probe after the fix (not committed) issued `SO_0008` over a
stored `SO_0007` on that face. That face has no permanent pin here, and
its pre-fix answer was not measured.
- Counters that an earlier release already seeded too low are not
rewritten by this change. The next collision on such a counter now
re-seeds it from the stored MAX, as on any other prefix.
- Patch round 2 (`f71f31b2`, seat edit): the branch merged `origin/main`
at `097ef802`, a clean merge with the same 4 files in the net diff. It
carries the sibling objectstack-ai#21208 (`862f12c0b`, `driver-turso`), and it re-ran
the `Type Check · source gates` lane that had timed out at `5b83098b`.
Every check is green on this head.

---
_Generated by [Claude
Code](https://claude.ai/code/session_017xfMoEjKUuSh2xYB8sCozp)_

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/l tests tooling

Projects

None yet

2 participants