Repository navigation
Commit f8178ff
fix(driver-memory): $contains on a multi-valued or JSON-stored field is membership, on every face (#20984)
Fixes #20874
Clause-②: yes (widening)
## What changes
`driver-memory` now answers `$contains` and `$notContains` on a
**declared JSON-stored field** by whole-element membership, the reading
`driver-sql` compiles on all three dialects. A field counts as
JSON-stored when it is `multiple: true`, a `multiselect` / `checkboxes`
/ `tags` field, or a `STRUCTURED_JSON_TYPES` member such as `json`. A
scalar text column keeps the case-exact substring test. The change
covers every face of the package:
- the query path's `$`-spelling (`normalizeFieldOperators`) and its AST
spelling (`convertConditionToMongo`): `find`, `count`, and every verb
that goes through `convertToMongoQuery`;
- the analytics (cube) face's mingo `$match`;
- the analytics face's SQL echo (`generateSql`), which now renders
SQLite's `json_each` membership construct for such a column, so the
echoed statement still returns the rows the chart was drawn from.
One rule backs all of them: `InMemoryDriver.filterContainsTest(object,
field, value)`. On a JSON-stored field it returns `{ $elemMatch: { $in:
members, $not: { $type: 'array' } } }`, and on any other field `{
$regex: filterSubstringPattern(value) }`. `$notContains` is `$not` over
that test, on both faces. The members come from the comparand's text,
read the way `driver-sql`'s `jsonMembershipCandidates` reads it: `'1'`
names the string or the number 1, `'1.50'` names 1.5, and `'true'` /
`'false'` / `'null'` name the string or the JSON literal.
The `$contains` docblock in `packages/spec/src/data/filter.zod.ts` lost
its stale "driver-memory DOES NOT ANSWER IT YET" bullet. Its dead
tracker link went with it (that card answers 404). The replacement text
states the measured status, and states that the other text operators
over a stored array are not ruled by that section.
## Measured on `origin/main` `f6ccca4a` before the change, and after
(HEAD `10656601`)
Fixture: `driver-sql`'s #17590 fixture plus a multi-valued lookup
`owners` (`['u1','u2']`, `['u10']`, `['u3','u1']`, `[]`).
| `where` | memory before | SQLite (`driver-sql`) | memory after |
|:--|:--|:--|:--|
| `{ owners: { $contains: 'u1' } }` | 1, **2**, 3 | 1, 3 | 1, 3 |
| `{ owners: { $notContains: 'u1' } }` | 4 (**2 dropped**) | 2, 4 | 2, 4
|
| `{ tags_: { $contains: 'red' } }` | 1, **2** | 1 | 1 |
| `{ nums: { $contains: '1' } }` (members are numbers) | **none** | 1 |
1 |
| `{ label: { $contains: 'red' } }` (scalar control) | 1, 2 | 1, 2 | 1,
2 |
| the cube face, the same filters | the query path's rows | n/a | the
query path's rows |
`engine.find` on a real `InMemoryDriver` with the nested-relation filter
`{ owners: { region: 'NA' } }` (owner `u1` NA, `u10` EU; this is PR
#20872's lowering): `d1, d3, d5` before and `d1, d3` after. Its `$not`
gave `d2, d4` before and `d2, d4, d5` after, which matches the rest
suite's SQLite rows. That was a one-off run (a throwaway probe file, not
committed): this package may not import the engine and the engine's
packages may not import this driver (`check:driver-memory-census`). What
is pinned instead is the driver input the engine sends, an `$or` of one
`$contains` per related id, which objectql's
`engine-nested-relation-lowering.test.ts` asserts.
## Mechanism hypotheses (order Zone 2)
- **H1 confirmed.** The `$contains` arm lowered to an escaped `$regex`,
and mingo applies a `$regex` to each element of an array value.
Reproduced on the fixture above before any edit.
- **H2: the fork is by the DECLARED field, not the row's runtime
shape.** Memory has the declaration (`valueShapes`, recorded by
`syncSchema` since the `$empty` work), and SQL forks on its JSON-column
registry, which is also filled from the declaration. The population is
the spec's JSON-stored classes: `STRUCTURED_JSON_TYPES` plus
`isMultiValueField`, the two halves `driver-sql`'s registry is built
from. The two readings DO give different rows on one fixture, so the
choice goes to the PM in the report's `open_questions`: a declared
`json` field holding the scalar string `'u1'`. SQLite answers no member,
because its constructs are array-only. A per-row-shape fork would answer
it by substring. The declared fork matches SQLite, and the contract text
says the question is "selected by the COLUMN", declared metadata.
Undeclared fields (an object never passed through `syncSchema`) keep the
substring reading, as `SqlDriver.isJsonColumn` answers `false` for a
table it was never told about.
- **H3 confirmed and pinned.** Number members answered nothing; `'1'`,
`'2'`, `'10'`, `'0'` and `'1.50'` now give SQLite's rows.
- **H4 confirmed: `$notContains` diverged.** It is the mirror arm of the
same defect, changed under os-dev rule 3's bounded in-place exemption.
All four conditions hold:
1. Same defect class.
2. Mechanical, with its shape pinned by SQL's `col IS NULL OR NOT (…)`:
`$not` over the test admits null and missing rows, pinned on both
fixtures.
3. `memory-driver.ts` is held by no other claim; the sibling `$exists`
card is kept off it by its own claim.
4. Same tests, no new gate.
**The claim's surface should be amended to include it.**
- **H5 confirmed.** The cube face borrowed `filterSubstringPattern` and
wrapped it in its own `$regex`, so it had the same defect. It now takes
the driver's whole test. The echo reads its member set off that same
test, so the chart and its echo cannot name different sets.
- **H6 confirmed.** See the `engine.find` run above.
## Compile-surface conclusions
| # | face | conclusion |
|---|---|---|
| 1 | `driver-sql` `applyFilterCondition` (`driver-sqlite-wasm`,
`driver-turso` local inherit it) | **already compliant**:
`applyJsonMembership` emits membership on JSON columns. Evidence: the
#17590 suite now carries a multi-valued lookup column and the `u1`/`u10`
case, green on SQLite here. Its live PostgreSQL/MySQL cells are
unprovisioned locally; the required live job runs the whole driver-sql
suite. The two inheriting drivers were not separately run. |
| 2 | turso `RemoteTransport.buildWhereSQL` | **out of scope** (another
package). By reading, its `$contains` arm emits `pushLike` (a GLOB
substring) on every column, JSON columns included, so remotely `u1`
would match `["u10"]` while the local transport answers membership. Not
measured. Reported as a finding. |
| 3 | service-analytics `read-scope-sql` `compileScopedFilterToSql` |
**out of scope** (another package). **Measured**: `{ owners: {
$contains: 'u1' } }` on a declared `lookup` + `multiple: true` field
compiles to `instr("t"."owners", ?) > 0` on SQLite and `LIKE '%u1%'` on
PostgreSQL/MySQL, a substring over the stored JSON text. This is an RLS
read-scope face, so it is reported as a security-relevant finding. |
| 4 | service-analytics `filter-normalizer` `lowerAnalyticsWhere` |
**out of scope**. By reading, it lowers `$contains` to the cube
`contains` operator, which the native SQL strategy renders as `LIKE`, a
substring. Reported. |
| 5 | `formula` `matchesFilterCondition` | **out of scope**. By reading,
the arm is `typeof actual === 'string' && actual.includes(v)`, so a
stored array never matches (fail-closed). Reported. |
| half | objectql `having-filter` | **out of scope by the claim**
(serial behind another card). Untouched. |
| unfrozen | `driver-memory` query path and cube face
(`memory-analytics.ts`), behind `filter-refusal.ts` | **changed** (this
PR). `filter-refusal.ts` and the `$exists` arm are untouched (sibling
card). |
| unfrozen | `driver-mongodb` `translateFieldOperators` | **out of
scope** (another package). By reading, it lowers `$contains` to a native
`$regex`, which MongoDB applies per array element: the same defect. Not
measured. Reported. |
## Why the shared pin is a mirrored literal table, not
`FILTER_TEXT_CASES` (order Zone 3, not taken)
`FILTER_TEXT_CASES` has no array column, so adding one changes the
fixture of all five enrolled drivers. `driver-mongodb` imports every row
of it and still carries the per-element defect, so its suite would go
red. The table's own rule 2 says rows join a driver's suite in the PR
that ends that driver's gap. Doing it here would also widen the spec
touch beyond the one declared docblock. A new sibling case-set would add
DEBT rows to a ledger that only goes down. So the new memory file
mirrors `driver-sql`'s #17590 fixture row for row and asserts the same
literal row sets. That is how the #17590 file already holds its three
dialect cells to one answer.
## Files
- `packages/drivers/driver-memory/src/memory-driver.ts`: the population
(`isJsonStoredField`), the members, the one test (`filterContainsTest`),
both query-path spellings.
- `packages/drivers/driver-memory/src/memory-analytics.ts`: the `$match`
rows take the driver's test; the echo renders membership
(`sqliteMembershipPredicate`).
-
`packages/drivers/driver-memory/src/memory-20874-contains-membership.test.ts`
(new): two fixtures. The first is the #17590 fixture plus `owners`. The
second holds the stored shapes SQLite decides: a scalar, an object, a
nested array, `[null]`, `[[null]]`, `[true, 1.5]`, `['']` and a NULL
row, each answer measured on `driver-sql`/SQLite first. Every case runs
on `find` in both spellings, on `count`, and on the cube's `$match`. The
cube's echo is EXECUTED on sql.js over the same rows. The file also
covers the nested-relation driver input and the fork.
-
`packages/drivers/driver-sql/src/sql-driver-17590-json-column-membership.test.ts`:
an `owners` multi-valued lookup column and the `u1`/`u10` case (the
claim's SQLite pin).
- `packages/spec/src/data/filter.zod.ts`: the one declared docblock.
- `.changeset/20874-memory-contains-membership.md`: **`minor`**, not the
`patch` the order suggested. `filterContainsTest` is a new public method
on the exported `InMemoryDriver`. It ships in `dist/index.d.ts` (the
existing `filterSubstringPattern` appears there as the positive
control), and the level ruling quoted in `pr-automation.yml` (WHICH
LEVEL) grades an additive widening of a published package's public
surface at least `minor`. `Clause-②: yes (widening)`: the new public
method widens the published surface; no filter key or operator is added.
- **Sweep, beyond the claim's listed surface.** The order's pin sweep
asks for every same-semantics statement to be flipped in one round. Each
item below is an edit to a statement this change makes false:
- `packages/rest/src/data-nested-object-door.test.ts` (a header comment
that cited the gap and the removed docblock clause);
- `.changeset/20802-nested-relation-filter-served.md`, one sentence of a
**pending** release note that said the in-memory driver still matches
per element.
## Confirmation requested: a pending release note is corrected
This PR changes `.changeset/20802-nested-relation-filter-served.md`, a
pending release note this PR did not add. This is the DELIBERATE
CORRECTION class `check-empty-changeset.mjs` names, so `Check Changeset`
stays **red** on purpose. It is not a required context. `skip-changeset`
must not be applied. The note's last sentence said:
> On the in-memory driver, a multi-valued relation's `$contains` still
matches a stored id by substring per element, so there an id that is a
substring of another stored id (`u1` inside `u10`) also matches; SQLite
and PostgreSQL match the element.
This PR makes that false, so it now reads:
> SQLite, PostgreSQL and the in-memory driver match the element of a
multi-valued relation, so an id that is a substring of another stored id
(`u1` inside `u10`) does not match it.
Please confirm the correction on this PR. If the release that consumes
that note ships before this PR lands, the edit should be dropped on
rebase.
## Tests (HEAD `10656601`)
- `pnpm --filter @objectstack/driver-memory typecheck` passes, and
`vitest run` gives **68 files / 1553 tests passed**. Both typecheck
programs include the new test file (`--listFiles`).
- The new file alone gives **113 passed**.
- `pnpm --filter @objectstack/driver-sql typecheck` passes (its program
includes the edited test).
`sql-driver-17590-json-column-membership.test.ts` gives **22 passed, 2
skipped**; the skips are the live PostgreSQL/MySQL cells, unprovisioned
here, NOT MEASURED locally.
- `pnpm --filter @objectstack/spec typecheck` passes. `spec build` and
then `check:generated` report "All 15 generated artifacts are up to
date".
**Reverse verification.** The code change was committed first. Then
`memory-driver.ts` and `memory-analytics.ts` were checked out at BASE
under an EXIT/INT/TERM restore trap. The landed mutation was verified on
disk before the run: `filterContainsTest` count 0, two
`escapeRegex(val)` arms back, and both files byte-equal to BASE.
The predicted direction was that the cases where per-element substring
and membership disagree go red and the agreeing ones stay green.
Measured: **69 failed / 44 passed**. The failures were every disagreeing
case, on all four faces, plus the executed echo, the relation-lowering
input and the fork pin. The `label` controls and the `redwood`, `ab`,
`u10` and `'0'` cases stayed green. The restore was proven by blob
hashes equal to HEAD (`d376dadb…` / `9de9198b…`), an empty `git diff
HEAD` and clean porcelain.
**Guard ablation** (`scripts/ablation-replace.mjs`, anchor 1 to 0).
Dropping `$not: { $type: 'array' }` turned exactly the nested-array
cases red: 12, the json `'u1'`, its complement and `'null'`, on all four
faces. The executed echo stayed green, since SQLite is the oracle there.
The file was restored to the HEAD blob with an empty `git diff HEAD`.
**Lint (narrowed, a measurement).**
1. Population: `eslint.config.mjs` lints
`**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}` minus `NEVER_LINTED`. The six
changed TS files are all in it; the two changesets are not.
2. `eslint --no-inline-config --format json` over those six files
reports 6 files, 0 errors, 0 warnings at `10656601`.
3. The config never enables type-aware linting (no
`parserOptions.project`, no typed rules; it says so itself), so this
diff cannot move any untouched file's verdict.
**Driver conformance ledger**: `node
scripts/check-driver-conformance.mjs` reads "50 covered cell(s), 0 in
the DEBT ledger, 0 exempt" both at `f6ccca4a` (before) and at `10656601`
(after).
**Gates**: `dispatch-gates --commands` was re-derived at `10656601`.
That gives 84 commands, six more than the dispatch list:
`engine-double-contract`, `objectql-double-limit`,
`query-options-erasure`, `type-check-coverage`, `type-check-debt` and
`where-matcher`. The run also covered the eight roster gates flagged as
sharing a directory with these paths. All exit 0 except:
- `check-empty-changeset`: exit 1, the deliberate correction above.
- `check:dual-build-cjs-loads` and `check:type-check-debt`: exit 3,
`PREREQUISITE NOT MET` (they need the whole workspace built). **NOT
MEASURED**; CI's Build/Lint jobs own those.
- `check:lean-entry-closure`: exit 3 at first. It passed after building
objectql.
## Acceptance notes (observed, not filed by this PR)
- The other text operators over a stored array are not ruled by the
membership section and still differ: `{ owners: { $startsWith: 'u1' } }`
gives `d1, d3, d5` on memory (per element) and no rows on SQLite (GLOB
over the serialized text). The spec docblock now says so.
- objectql `engine.ts`'s delete-probe docblock (the `$contains`
pushdown) still says every backend answers a substring SUPERSET. That is
stale for `driver-sql` since #17590 and for memory after this PR. The
code stays correct because it narrows exactly through
`storedReferenceIncludes`. Comment only, no carrier.
- A field `driver-memory` holds no declaration for keeps the substring
reading, and mingo still applies that `$regex` per element of an array
value. This is the deliberate counterpart of `driver-sql`'s
`isJsonColumn` returning false for an unknown table.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent a5bce40 commit f8178ff
8 files changed
Lines changed: 676 additions & 43 deletions
File tree
- .changeset
- packages
- drivers
- driver-memory/src
- driver-sql/src
- rest/src
- spec/src/data
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | | - | |
| 28 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
| 23 | + | |
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| |||
238 | 238 | | |
239 | 239 | | |
240 | 240 | | |
241 | | - | |
242 | | - | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
243 | 244 | | |
244 | 245 | | |
245 | 246 | | |
246 | 247 | | |
247 | 248 | | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
248 | 256 | | |
249 | | - | |
| 257 | + | |
250 | 258 | | |
251 | 259 | | |
252 | 260 | | |
| |||
335 | 343 | | |
336 | 344 | | |
337 | 345 | | |
338 | | - | |
339 | | - | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
340 | 349 | | |
341 | 350 | | |
342 | 351 | | |
343 | 352 | | |
344 | 353 | | |
345 | 354 | | |
346 | 355 | | |
347 | | - | |
348 | | - | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
349 | 359 | | |
350 | 360 | | |
351 | 361 | | |
| |||
386 | 396 | | |
387 | 397 | | |
388 | 398 | | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
389 | 445 | | |
390 | 446 | | |
391 | 447 | | |
| |||
537 | 593 | | |
538 | 594 | | |
539 | 595 | | |
540 | | - | |
541 | | - | |
542 | | - | |
543 | | - | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
544 | 610 | | |
545 | 611 | | |
546 | 612 | | |
| |||
1595 | 1661 | | |
1596 | 1662 | | |
1597 | 1663 | | |
| 1664 | + | |
1598 | 1665 | | |
1599 | 1666 | | |
1600 | 1667 | | |
1601 | | - | |
| 1668 | + | |
1602 | 1669 | | |
1603 | 1670 | | |
1604 | 1671 | | |
| |||
1663 | 1730 | | |
1664 | 1731 | | |
1665 | 1732 | | |
| 1733 | + | |
1666 | 1734 | | |
1667 | 1735 | | |
1668 | 1736 | | |
1669 | 1737 | | |
1670 | 1738 | | |
1671 | 1739 | | |
| 1740 | + | |
| 1741 | + | |
| 1742 | + | |
| 1743 | + | |
| 1744 | + | |
| 1745 | + | |
1672 | 1746 | | |
1673 | 1747 | | |
1674 | 1748 | | |
| |||
0 commit comments