Repository navigation
fix(formula,plugin-security): the cross-class field-comparison refusal leads with its remedy, so REST callers read the fix (#20869) - #20972
Conversation
…remedy The matcher's refusal now opens with the fix and fits the REST client message bound whole (494 characters); the explain engine's copy puts the same remedy before its unbounded subject and diagnostic. Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude <noreply@anthropic.com>
…edy on the wire The /data insert and find doors and POST /security/explain, through the real security layer on driver-sql: each wire message carries its producer's remedy, a long-names fixture keeps the remedy under the bound, and a short refusal of another class reaches the wire unchanged. Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude <noreply@anthropic.com>
…rst refusal Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude <noreply@anthropic.com>
📓 Docs Drift Check2 anchor(s) derived from 2 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run. What this run could not see
Coarse fallback — 22 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 2aa7dc325e6a4db2dfb73a713af1db3ca4d6debc && git checkout 2aa7dc325e6a4db2dfb73a713af1db3ca4d6debc
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 4957ee5ef0e660fc9ee4525d83f13aa32ef8e969 dc440bff6bf9c731ce7f315ac8015de9624370b1 && git checkout -B drift-repro 4957ee5ef0e660fc9ee4525d83f13aa32ef8e969 && git merge --no-ff dc440bff6bf9c731ce7f315ac8015de9624370b1
node scripts/docs-audit/affected-docs.mjs --json 4957ee5ef0e660fc9ee4525d83f13aa32ef8e969 |
Contract reviewServed-tier: Inputs, and nothing else: card #20869 (body; comments 5912895835, 5920895653, 5921682938), #20355 with its comments (the withheld posture), #5423, PR #20972 (body, file list, net diff against ① Derived judgments
② Semver level
③ Boundary flags
Implemented-by: VERDICT: PASS Generated by Claude Code |
…alued field by membership, as its where twin does (objectstack-ai#21004) Fixes objectstack-ai#20873 Clause-②: no ## What changes The per-aggregation `filter` (`engine.aggregate({ aggregations: [{ …, filter }] })`, and so `POST /api/v1/data/:object/query`) is evaluated by the engine's own walker, `matchesAggregationFilter` in `packages/objectql/src/having-filter.ts`. Its `$contains` arm failed every value that was not a string, so a stored array never matched, and its `$notContains` arm passed every such value, members included. On a DECLARED JSON-stored field both arms now ask MEMBERSHIP, the reading `FILTER_OPERATORS`' `$contains` docblock (`@objectstack/spec`) declares and `where` already gives on every SQL dialect (`SqlDriver.applyJsonMembership`): - `$contains: v` holds when `v` names an element of the stored array. A member stored as a JSON number or boolean is named by its text (`'1'` names `1`, `'1.50'` names `1.5`, `'true'` names `true`, `'null'` names `null`). That is the candidate set `driver-sql`'s `jsonMembershipCandidates` binds on every dialect. Array-only, as the SQL constructs are. - `$notContains: v` is its exact complement, and a row with no value still satisfies it (objectstack-ai#5298), as `col IS NULL OR NOT (…)` does in SQL. - A scalar text column keeps the substring test, unchanged. So does `having`. Files: `having-filter.ts` (the two arms, `storedArrayHasMember`, `declaredJsonStoredFields`, and an optional `jsonStored` set threaded through `matchesHaving` / `matchesAggregationFilter`), plus `in-memory-aggregation.ts`. That file is the one place the engine hands the object's declared field map to the per-aggregation filter, so it reads the declared set once per call beside `declaredFieldClasses`. `having-filter.ts` is not on objectql's published entry points. `in-memory-aggregation.ts` is (`applyInMemoryAggregation` and `bucketDateValue`, from `index.ts` and `core.ts`), and neither exported signature changes; the declared set is threaded through the internal `aggregateBucket` only. ### The card's table, through the REST door, before and after Measured with a real `SqlDriver` on SQLite and on a live PostgreSQL 16.14. Rows: `d1 ['u1','u2']`, `d2 ['u2']`, `d3 ['u3','u1']`, `d4 []`, `d5 ['u10']`, `d6 null`. Base `212d613c`, head `90ba78d9`. | query | `where` twin | per-aggregation `m`, base | `m`, head | |:--|:--|:--|:--| | `owners $contains 'u1'` (the card) | 2 | 0 | 2 | | `owners $contains 'u10'` | 1 | 0 | 1 | | `owners $notContains 'u1'` | 4 | 6 | 4 | | `tags $contains 'red'` (`d3` holds `['redwood']`) | 2 | 0 | 2 | | `tags $notContains 'red'` | 4 | 6 | 4 | | `$or` of `$contains` u1 / u3 (the any-of spelling objectstack-ai#7398's refusal prescribes) | 2 | 0 | 2 | | `$not` over `owners $contains 'u1'` | 4 | 6 | 4 | | `title $contains 'u1'` (text, the control) | 3 | 3 | 3 | On the in-memory driver the per-aggregation `m` is the same evaluator's answer, also 2 now. Memory's own `where` answers 3 for the card at this base (`d5` too, by a per-element substring). That face belongs to objectstack-ai#20874 (in flight), whose branch (`10656601`) moves it to membership and pins `d1, d3`. ## The fork: by the DECLARED column (Zone 2 H2) The fork reads the declaration (`STRUCTURED_JSON_TYPES` or `isMultiValueField`), never the row. That is the contract's sentence: "One operator, two questions, selected by the COLUMN rather than by the caller". It is also `SqlDriver.isJsonColumn`'s population (built from the same two spec sets) and objectstack-ai#20874's `isJsonStoredField`, character for character. Measured: on every fixture reachable through the public doors, the declared reading and a value-shape reading select the same rows. - `$contains` / `$notContains` on a declared structured-JSON field is refused before any row is read: the engine's text-operator declared-type door, `INVALID_FILTER` 400, in `where` and in the per-aggregation filter alike, on all three backends. - A multi-valued field's `find()` value is an array or `null` on memory, SQLite and PostgreSQL alike. The write door wraps a scalar: `'u1'` is stored as `['u1']` on memory and SQLite. The two readings differ only on rows a direct caller hands the walker: a declared multi-valued column holding a scalar string, or an undeclared column holding an array. There the declared reading gives what SQL `where` gives (no member; the substring reading), and a value-shape reading would not. Both cases are pinned. No `open_questions` fork results. ## Zone 2 hypotheses, measured - **H1 — confirmed.** The arms were as hypothesised; the card's table reproduced on all three backends (`m: 0`). - **H2 — declared column**, above. - **H3 — `$in` / `$nin` left as they are.** - `where: { owners: { $in: ['u1','u9'] } }` is refused `INVALID_FILTER` 400 on SQLite and PostgreSQL (the objectstack-ai#7398 JSON-column gate). Memory's `where` answers `d1, d3`. - The drivers disagree and SQL refuses, so no membership semantics are invented here. - The per-aggregation answer stays `m: 0` for `$in` and `m: 6` for `$nin`, a 200 where `where` is a 400. That is reported as an out-of-scope finding, not pinned. - **H4 — `having`.** - A `groupBy` on a multi-valued field is refused 400 on all three backends, and on a structured-JSON field too. - The one aggregated column that can still hold a stored array is a `min` / `max` over a multi-valued field. That is answered three ways: an array on memory, the serialized TEXT on SQLite's native aggregate, `DATABASE_ERROR` 500 on PostgreSQL. So there is no single `where` answer to hold `having` to, and `having` is not handed the declared set. - What is pinned: `having` `$contains` on a `groupBy` text projection keeps substring, on SQLite and PostgreSQL at the REST door and on the engine level. - **H5 — confirmed, so the mirror arm moved under os-dev rule 3's bounded in-place exemption.** Base per-aggregation `$notContains 'u1'` counted 6 where `where` counts 4. - All four conditions hold. Same defect class, same arm pair. The shape is pinned by `applyJsonMembership`'s complement. No other claim holds `having-filter.ts`: objectstack-ai#20822 group 3 is unclaimed, and objectstack-ai#20981 is filed bare. Same gate family. - The NULL row `d6` is counted, as SQL's `col IS NULL OR NOT (…)` counts it. - The claim's file surface does not name this arm. PM: please amend it, together with `in-memory-aggregation.ts` and the two test files. - **H6 — no importable predicate.** `driver-sql`'s `jsonMembershipCandidates` and `driver-memory`'s `containsMemberCandidates` (landed by PR objectstack-ai#20984 after this branch's merge base) are both module-private in driver packages, which objectql does not depend on. So `storedArrayHasMember` is the third copy of the rule on `main`; see Acceptance notes. ## Compile-surface conclusions | # | face | conclusion | |:--|:--|:--| | 1 | `driver-sql` `applyFilterCondition` | **already compliant (evidence)** — `$contains` / `$notContains` on a JSON column go through `applyJsonMembership`; the `where` twin numbers in the table above are this face, measured on SQLite and PostgreSQL 16.14. `driver-sqlite-wasm` and `driver-turso` local inherit it (not measured separately). | | 2 | turso `RemoteTransport.buildWhereSQL` | **out of scope (reason)** — an independent compiler this card does not touch. Read at `212d613c`: its `$contains` / `$notContains` arms go `pushLike` (substring over the stored text) with no JSON-column fork. Not measured (no remote libsql here). In the out-of-scope finding below. | | 3 | service-analytics `compileScopedFilterToSql` | **out of scope (reason)** — a different compiler. Measured function-level at `212d613c`: `{ owners: { $contains: 'u1' } }` compiles to `instr("t"."owners", ?) > 0` on SQLite, which admits a row holding `["u10"]`. On PostgreSQL it compiles to `"t"."owners" LIKE ? ESCAPE ?` over a json column. In the out-of-scope finding below (an RLS read scope). | | 4 | service-analytics `lowerAnalyticsWhere` | **out of scope (reason)** — it lowers the analytics `where` to a `FilterCondition` and adds no `$contains` reading of its own. The ObjectQL strategy hands that to the driver (face 1). The native SQL strategy maps `contains` to the substring LIKE shape (`native-sql-strategy.ts`, read, not measured), in the same finding as face 3. | | 5 | `formula` `matchesFilterCondition` | **out of scope (reason: fenced; PR objectstack-ai#20972 landed on this file during this run)** — its arm is `typeof actual === 'string' && typeof v === 'string' && actual.includes(v)` (unchanged by objectstack-ai#20972). Measured: `['u1','u2']` → false, `['u10']` → false, `'u1 memo'` → true. In the out-of-scope finding below. | | half | objectql `having-filter` | **changed** — the per-aggregation filter, as above; `applyHaving` / `matchesHaving` without a declared set unchanged (H4). | | unfrozen | `driver-memory` / `driver-mongodb` | **out of scope (reason: fenced, objectstack-ai#20874 / objectstack-ai#20897 in flight).** Memory measured above, and objectstack-ai#20874's fork matches this one. Mongo's `translateFieldOperators` compiles `$contains` to a bare `$regex`, which MongoDB applies per array element (per-element substring). Read, not measured. | ## Tests - `pnpm --filter @objectstack/objectql exec vitest run --project local --maxWorkers=2 src/engine-aggregate-filter-array-membership.test.ts` — 30 passed. The card's rows with the rows themselves, empty table, per group, `having` control, the member-text reading (number / exponent / boolean / null / non-JSON-number spellings / nested / object / scalar), the declared fork, the declared population. - `OS_TEST_POSTGRES_URL=… pnpm --filter @objectstack/rest exec vitest run --project local --maxWorkers=2 src/aggregation-filter-array-membership.test.ts` — 18 passed (9 SQLite, 9 live PostgreSQL 16.14), 9 named skips (MySQL). Each row runs beside its live `where` twin, populated and empty. - objectql whole `local` project on the merged head `90ba78d9`: 349 files, 6851 tests passed; `repo` project 1 file / 5 passed. - REST aggregation-adjacent files on `90ba78d9` with the PostgreSQL cell live: 9 files, 106 passed, 34 skipped. - `pnpm --filter @objectstack/objectql typecheck` and `pnpm --filter @objectstack/rest typecheck`: green. Both new test files are in their package's test program (`tsc -p tsconfig.test.json --listFiles`). ### Reverse verification Each leg ran through `scripts/ablation-replace.mjs` (anchor hits proven on disk; restore proven blob == HEAD and `git diff HEAD` empty), from the committed change. - **A1** — the `$contains` arm put back to the substring test: 15 of 30 red (every membership `$contains` row, the member-text rows, the declared-fork row); the `$notContains` rows and controls green, as predicted. - **A2** — the `$notContains` arm put back: 3 red (its two rows and the complement row). - **B** — the REST suite reads objectql through `dist/`. `declaredJsonStoredFields` was emptied, objectql rebuilt, and `ablation-dist-preflight.mjs` found the marker in 4 built files. 12 red: the 6 membership rows on each of SQLite and PostgreSQL. Text control, `having` and empty-table rows stayed green. Restore leg: rebuilt, marker absent from all 14 built files, tree clean, 18 passed. ### Driver conformance ledger `node scripts/check-driver-conformance.mjs`: before (`212d613c`) "50 covered cell(s), 0 in the DEBT ledger, 0 exempt"; after (`90ba78d9`) the same. ### Gates - `node scripts/pm/dispatch-gates.mjs --commands` re-derived with no paths at `90ba78d9` gives 63 commands, all run, exit codes recorded to disk. `--ran` reconciliation: 63 derived, 61 run (all exit 0), 2 NOT MEASURED, 0 unrun. - NOT MEASURED: `check:dual-build-cjs-loads` and `check:type-check-debt`. Both are PREREQUISITE NOT MET (exit 3): they need the whole-workspace build `lint.yml` performs first, and 42 / 5 packages have no `dist/` in this worktree. - `check-engine-split-ratio` first refused on the shallow checkout. It was green after a deepen to its window (`git fetch --shallow-since=2026-06-26 origin main`). - Lint, a declared narrowing over the four touched TS files at `90ba78d9`: `eslint --no-inline-config --format json` reports 4 files, 0 errors, 0 warnings. - Each file maps to a config (`--print-config`). - `eslint.config.mjs` never enables type-aware linting (no `parserOptions.project`, no typed rules; its own lines 327-328 say so), so this diff cannot move an untouched file's verdict. ## Acceptance notes - **The member rule now has three copies on `main`.** - `storedArrayHasMember` restates `driver-sql`'s `jsonMembershipCandidates` as a predicate. `driver-memory`'s `containsMemberCandidates` (PR objectstack-ai#20984) is the other JS copy. - None can be imported by the others. The one shared home would be `@objectstack/spec/data`, beside `asciiCaseInsensitiveContains` and `isEmptyFilterValue`, the value-level filter rules every JS face already reads from there. - Carrier: the convergence item recorded on objectstack-ai#20987. - **No shared conformance kit for stored-array membership.** - `FILTER_TEXT_CASES` has no array rows. The membership fixtures are literal per package: `sql-driver-17590-json-column-membership.test.ts`, objectstack-ai#20874's `memory-20874-contains-membership.test.ts`, and the two files here. These use the same `u1` / `u10` / `redwood` disagreement rows. - A spec `*_CASES` kit beside `FILTER_TEXT_CASES` would let all three faces be driven by one table. - Here the cross-face invariant is held by running each row beside its live `where` twin. - **The memory cell is engine-level.** It runs over the read shape `find()` presents (measured identical on all three backends). A real `InMemoryDriver` consumer would need a ruled entry in the `check:driver-memory-census` ledger. - **The PostgreSQL / MySQL REST cells are a named skip in CI.** No job provisions `OS_TEST_POSTGRES_URL` / `OS_TEST_MYSQL_URL` for `packages/rest`, the same note `data-group-by-json-door.test.ts` carries. The PostgreSQL cell's local run is above. - **Out-of-scope findings, for the seat to route** (full evidence in the report on the card): - per-aggregation `$in` / `$nin` on a multi-valued field answer 200 (`m: 0` / `m: 6`, the latter counting the rows it was asked to exclude) where `where` is a 400 on the SQL family, and memory's `where` answers membership; - the `$contains` membership contract is not answered on the turso remote transport, the service-analytics SQL compilers (an RLS read scope over-reaches), `driver-mongodb` and `formula`; - `min` / `max` over a multi-valued field: three answers (array / serialized text / PostgreSQL 500); - `$startsWith` / `$icontains` on a multi-valued field: PostgreSQL `where` 500, SQLite over the serialized text, memory per element. --- _Generated by [Claude Code](https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
Fixes #20869
Clause-②: no
What this changes
The refusal of a field-to-field comparison across comparison classes (
INVALID_FILTER/ 400) now opens with its remedy, on both of its producers. The REST door bounds a 4xx message by cutting its tail (CLIENT_MESSAGE_MAX = 500: 499 characters plus an ellipsis). Both producers wrote the remedy last, so no caller ever read it. The bound is not touched; that is #5423's decision. The producers change.packages/formula/src/matches-filter.ts,crossFieldClassError. The remedy sentence comes first, byte-identical to the one the message ended with. The diagnosis is shortened so the whole message is 494 characters and reaches the wire whole. Order: the remedy; what is refused (two columns with no shared class, and the classes); why it is refused; why the columns are withheld. It still names no column, operator or policy; the columns travel on the error's symbol key for the server log only.packages/plugins/plugin-security/src/explain-engine.ts,crossFieldRefusalForExplain. The remedy sentence, also byte-identical, moves before the subject and the diagnostic. Those have no length bound: object, field and policy names declare no maximum (SnakeCaseIdentifierSchema, the objectnameand the RLS policynameare regex-only), and the subject lists every refused policy. So no subject-first order keeps a trailing remedy for every policy; at index 0 it survives any length. The reason drops one redundant clause ("instead of judging a record": the same sentence already says explain "reports no verdict").Code, status and trigger are unchanged. Only the text moves and shortens.
Measured through the real handlers
ObjectQL on driver-sql (better-sqlite3),
SecurityPlugin,RestServerroute handlers, and a policyrecord.status != record.amount. "Remedy at" is the index where the remedy sentence starts.013f97df93)POST /data/:object(insert: the RLS write check)GET /data/:object(find)POST /security/explain, short namesPOST /security/explain, 60-character namesGET /data/:objectwith{ title: { $bogus: 1 } }A find never carries the matcher's message. Only the RLS write check (
security-plugin.ts,satisfiesCheck) and explain (matchUnderDeclaredColumns) handmatchesFilterConditionthe declared columns its class rule reads; the other runtime caller (objectqlhaving-filter.ts) passes none. On/dataa find answers driver-sql's own read refusal of the same comparison, 383 characters, which already reached the wire whole and states the rule ("compared as the same type class"). The matcher's text reaches/dataon an insert or an update. So the/datapin covers both: the insert carries the matcher's remedy, and the find carries its read refusal whole.Pins
packages/rest/src/cross-class-refusal-remedy-on-the-wire.test.ts(new, the real stack, both envelopes this family speaks):/datainsert: the wire message starts with the matcher's remedy, is under the bound, equals the thrown message, and names neither column;/datafind: the read refusal reaches the wire whole and states the same-class rule;POST /security/explain: the wire message starts with the explain remedy, is within the bound, and names the policy;packages/formula/src/matches-filter-cross-field-class.test.ts: the message starts with the remedy, is under 500, is the same for long column names, and keeps the clause order.packages/plugins/plugin-security/src/explain-cross-class-refusal.test.ts: every refused cell (5 predicates, read / update / delete, SQLite and sqlite-wasm) asserts the message starts with the remedy.packages/plugins/plugin-security/src/rls-check-cross-class-field-refused.test.ts: the pinned opening moves with the text.Red, then green. The new REST pin reads the producers through their
dist/(theresttoplugin-securitypair is unaliased and registered incheck:test-source-alias). At2cfaa6522f, against producer builds from BASE013f97df93(new-text markers: 0 in both dists): 3 failed (insert, explain, long names) and 2 passed (find, control). After rebuilding both producers from2cfaa6522f(markers: 1 in each): 5 passed.Verification (head
dc440bff6b, after mergingorigin/mainatf6ccca4a44)5fde18e296(before the merge):@objectstack/formula:test42 files, 1241 passed;typecheckOK, test-layer debt held;@objectstack/plugin-security: 149 files, 3227 passed, 23 skipped (the PostgreSQL legs, no server);typecheckOK;@objectstack/rest:--project local245 files, 4875 passed, 114 skipped;typecheckOK, 0 test-layer errors.dc440bff6b(the merge brought commits intorest): the formula pin file 23 passed; the three plugin-security cross-class files 150 passed, 23 skipped;@objectstack/rest--project local246 files, 4890 passed, 114 skipped.node scripts/pm/dispatch-gates.mjs --commandsatdc440bff6bderives 65 commands; all 65 exit 0.--ranreconciliation: "65 derived, 65 run, 0 NOT-MEASURED, 0 UNRUN". Three of them (check:dual-build-cjs-loads,check:i18n,check:type-check-debt) first exited 3 (PREREQUISITE NOT MET: nothing measured). After the fullturbo run build --filter='./packages/*' --filter='./packages/*/*'(71 tasks), all three exit 0.dc440bff6b. ESLint's ownisPathIgnoredandcalculateConfigForFileput 6 of the 7 changed paths in its population; its config ignores the changeset.md.lintFilesover those 6 withallowInlineConfig: false(thelintscript's--no-inline-config), JSON formatter: 6 files, 0 errors, 0 warnings. The config enables no type-aware linting (noparserOptions.project, no typed rules), so this diff cannot move a verdict on an untouched file. The repository-widepnpm lintbelongs to CI.Acceptance notes
fieldReferenceUnsupportedError: the driver has no field-to-field lowering at all). It is 538 characters from source, so the bound cuts it. Its remedy ("Compare against a literal value instead.") ends at 410 and survives; the cut drops the end of its withholding sentence. Not edited here. Source reading plus the bound's arithmetic; not measured through a REST door, because no MongoDB server was available.packages/rest/src/security-explain-envelope.test.tsstill builds its matcher and explain refusals by hand, with the old opening, and says neither@objectstack/formulanor@objectstack/plugin-securityis a dependency ofrest.plugin-securityis a devDependency now. The hand-built text is a fixture, not a pin of either producer, and the route reads only its code and status. Left as is.Generated by Claude Code