feat(service-analytics,driver-memory): the shared filter lowering at the analytics seams and the cube face's new door (#5930 step 3) - #20857
Conversation
… at the analytics seams and the cube face's door (red) Pins first, before the fix. Three seams, one file each: - read scope (compileScopedFilterToSql entry): a bare-day upper bound on a declared datetime answers SqlDriver.find's rows, for a caller filter and for an RLS using bound in both spellings; the last supported day; a declared date control; the typed scope as SQL. - analytics where / preview door: the where -> tree face compiles the lowered condition (typed on datetime members), the ObjectQL hand-off and echo carry it, and the draft preview evaluates it (with $null). - the cube face's new door: the two shared comparand doors refuse an undefined comparand on both exits; the face compiles the lowered condition and the $or / $null vocabulary, and still refuses $not. Red on this commit for exactly those rows. Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
… at the analytics seams and the cube face's new door ADR-0053 D-D1 as amended: one shared FilterCondition -> FilterCondition lowering (lowerFilterCondition, @objectstack/spec/data), run at each analytics seam after the comparand doors and after filter-token resolution. - analytics where / preview door: normalizeAnalyticsFilterTree lowers the condition the door admitted before buildNode reads it (F10), with a REQUIRED column-type reader: both strategies pass declaredDatetimeLowering (a member is datetime when its column is declared so); member readers and a context with no hook pass NO_DATETIME_COLUMNS. The draft preview lowers after normalizeWhereComparands with NO_DATETIME_COLUMNS (F11) and evaluates $null. - read scope: compileScopedFilterToSql lowers at its entry, after the placeholders resolve, typed by the declaredValueShape both consumers already hand it. - the cube face (F5): normalizeFilters runs assertListComparandShapes and normalizeFilterComparandTypes, then the lowering (type-blind: the face cannot read declarations), then its own gate on the lowered condition. It compiles $or (as a disjunction entry both exits render) and $null, and still refuses $not. Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
…red condition; spell nested relations dotted at the where door The analytics where door now spells a nested-relation field spec as the dotted members fieldLeaves compiles, before the shared lowering reads the condition, so a guard the lowering lays inside a $not names the member the leaf binds and never the relation key itself. Existing pins updated to the lowered structure, rows unchanged: - SQL-text / tree / engine-where shape pins for $ne, $nin, $notContains and $not now show the lowering's NULL guard around each face's own interim copy of it (idempotent in rows until the copies' deletion cards); every row-count assertion beside them is unchanged. - the where-door tests pass the required column-type reader (NO_DATETIME_COLUMNS). - the cube face's refusal and vocabulary pins: $or and $null are compiled, a $between reaches the face as its two bounds, and each is held to find()'s rows; $not, $startsWith, $endsWith and $empty stay refused. Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
…face's new door Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
…alytics-seam-lowering
📓 Docs Drift CheckThis PR changes 2 package(s): 3 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:
⛔ 2 release-owned page(s) also name something this change touched. These are read-only:
What this run could not see
Coarse fallback — 15 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 67ba542ae915d3adb3816698e978508ae74522fe && git checkout 67ba542ae915d3adb3816698e978508ae74522fe
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 660a9b247e824f7747d63f3b778dccc9cb4751d6 edb3e2de482cf74a6d1c438b4ecd354fec64e799 && git checkout -B drift-repro 660a9b247e824f7747d63f3b778dccc9cb4751d6 && git merge --no-ff edb3e2de482cf74a6d1c438b4ecd354fec64e799
node scripts/docs-audit/affected-docs.mjs --json 660a9b247e824f7747d63f3b778dccc9cb4751d6
|
…ened vocabulary The claim's surface gained this file (amendment 5911719808) for exactly this change. Three pins encoded the cube face's vocabulary and shape from before the ruled widening; each is re-spelled, none removed: - every $not case must still refuse; the $or cases moved to the answered column, which the file's case-by-case invariant holds to find()'s rows; - $null joins the declared-operator probe roster; - the $notContains pipeline dump shows the lowering's NULL escape. Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
Contract reviewServed-tier: Read at 2026-09-30T13:45Z: card #20810 (its body and all six comments: the triage grade, the claim and its surface amendment, the two dev reports and the ACCEPT), PR #20857 (its body, its 41-file list, and the net diff against main from merge base c90f9fb: 1490 lines added, 234 removed), and the 41 check-runs on this head. Origin main at the reading (72f8c38) touches none of the 41 files after the merge base, so the net diff is the branch's whole delta. No spec path and no governed path in the file list. Check-runs on this head, latest run per name: every run has completed; none was still in progress at the reading. The seven required contexts all conclude success: Lint and Repo Gates, TypeScript Type Check, Test Core (and its six shards), Dogfood Regression Gate (and its three), Build Core, Temporal Conformance (live PG + MySQL), Governed Surface Queue Guard. Check Changeset concludes success in both generations. A second pr-automation generation ran on the PR-body edit; in it Auto Label and Check PR Size conclude skipped (their first-generation runs concluded success; both are advisory). Console Pin Gate, Build Docs and the packed-tarball smoke are skipped by their filters. The Docs Drift Check (advisory) lists three hand-written pages through the string literal account.region on a changed comment line; the nested-relation spelling compiles to the same member, the same SQL and the same rows as before, so none of the three pages moves. ① Derived judgmentsJudged against ADR-0053 D-D1 as amended in place at this head (items 1 to 10) and docs/design/predicate-compilation-convergence.md sections 3.4, 3.6 and 4.1. The shared lowering (packages/spec/src/data/filter-lowering.ts, step 2's) is untouched. Every seam calls it after the shared comparand doors, and every path that reaches the analytics door or the read scope has its filter tokens resolved upstream (resolveQueryTokens for the query's where and windows, resolvedDatasetScopeGetter for the dataset scope's filter and measure filters, DatasetExecutor.resolveSelectionTokens for the preview; the read scope resolves placeholders as its first act). Item 3 holds at every position this diff adds.
② Semver level
③ Boundary flagsThe dev report lists
Implemented-by: VERDICT: PASS Generated by Claude Code |
Fixes #20810
Clause-②: yes (narrowing)
#5930 step 3, under ADR-0053 D-D1 as amended by #20754: the one shared
FilterCondition → FilterConditionlowering (lowerFilterCondition,@objectstack/spec/data, landed in step 2 as #20794) now runs at the three analytics seams, after the comparand doors and after filter-token resolution, and the two faces that could not compile its output now can.packages/spec/src/**, the interim window arms and #20807's position are untouched.Named gap: three pins outside the claim's file surface
packages/drivers/driver-memory/src/memory-driver-filter-logic-conformance.test.tsholds three pins of the cube face's PRE-change vocabulary and shape, and it is outside the claim'smemory-analytics*.test.tssurface, so this branch does not edit it. They are red here, and thedriver-memoryTest Core shard is red until they move:$orcase refuses. The face compiles$ornow; the same file's case-by-case invariant (agree withfind()or refuse) stays green over every$orcase.$null, which joined the face's table.$notContainspipeline-dump row pins the un-lowered$match; the lowering's NULL escape now wraps it.The three-row patch (15 changed lines) is prepared and was verified on a copy of the file: 125 of 125 tests pass. It lands once the file is added to the surface.
Per seam
bbe03f406wheredoor, F10 (where→ tree, both strategies)normalizeWhereComparandsfilter-normalizer.ts:2218, reached throughlowerAnalyticsWhere:2244; the lowering sits innormalizeAnalyticsFilterTree:2356, the one compile entry both spellings reachwhere-door-shared-lowering-seam.test.ts, 18 rowspreview-evaluator.ts:675, right after itsnormalizeWhereComparandscallcompileScopedFilterToSqlread-scope-sql.ts:747, after placeholder resolution:764, beforecompileNode:769read-scope-shared-lowering-seam.test.ts, 12 rowsnormalizeFiltersmemory-analytics.ts:1585, ahead of its gate:1592memory-analytics-shared-lowering-door.test.ts, 13 rowsnormalizeAnalyticsFilterTreeEvery ablation mutated the committed file through
scripts/ablation-replace.mjs(anchor hit once, blob moved), ran the pins, and restored: blob equal to HEAD andgit diff HEADempty, each time. The pins import their subjects by relative path, so vitest readssrc/and no build sits between the mutation and the reading.What each seam does
compileScopedFilterToSqllowers the scope right after its placeholders resolve. Column-type scope (item 7): thedeclaredValueShapeoption both consumers already pass, so a declareddatetimecolumn is rewritten and adate,timeor text column compiles byte-identical; no declarations handed in reads no column asdatetime. The shared comparand doors still judge the scope as written after compilation (service-analytics: the NativeSQL read-scope compiler and the/analytics/sqlecho compile two scope shapes the shared comparand faces refuse (plain-object comparand under$eq, null member in$in): one scope, two answers across faces #20018's order); the lowering never refuses, so no verdict moves.wheredoor (F10).normalizeAnalyticsFilterTreelowers what the door admitted beforebuildNodereads it. Its column-type reader is now a REQUIRED argument: both strategies passdeclaredDatetimeLowering(a member isdatetimewhen its column is declared so, through thedeclaredFieldTypehook, asked of the same target each strategy compiles against); a context without the hook, and the member-only readers (assertWhereFields, the cross-object view), passNO_DATETIME_COLUMNS.lowerAnalyticsWherestays un-lowered: its other readers (ad-hoc cube dimension minting, the routing detectors) read the authored condition.wheredoor. The shared lowering has no reading of the nested-relation spelling ({ account: { region: 'NA' } }: accepted by the schema, refused by the engine, flattened only by this door). Inside a$notit readaccountas a column and guarded it. The door now spells nested relations as the dotted membersfieldLeaveshas always compiled them to, before the lowering reads the condition, so the guard lands onaccount.region.normalizeWhereComparandswithNO_DATETIME_COLUMNS: drafted rows carry no schema, and a type-blind rewrite would move one cell away from the typed drivers ($lteon the last supported day over a non-temporal value that sorts above it); its ownlteBoundcopy keeps the whole-day rule until its deletion card. It now evaluates$null.normalizeFiltersrunsassertListComparandShapes, thennormalizeFilterComparandTypes(its return is the condition read from there on), then the lowering (type-blind: the face reaches declared types only as a storage-form conversion, and its own copy andfind()are type-blind already), then its own gate on the lowered condition. It compiles$or(a disjunction entry both exits render, with the 空组合子在同仓有两个对立答案:五个后端归约成布尔单位元,service-analytics 的两个编译器 fail-closed 抛错 —— #5239 的一致性表四条因此进不了表 #5322 identities) and$null;$not,$startsWith,$endsWithand$emptystay refused.Every corrected answer
Read scope (
compileScopedFilterToSql: the NativeSQL statement's scope and the/analytics/sqlecho's), each nowSqlDriver.find's rows on the same filter:{ signed_at: { $lte: '2026-07-28' } }on a declareddatetimeover rows at 10:00Z on 07-27, 07-28, 07-29 and a row with no value: was 07-27, now 07-27 and 07-28 (< '2026-07-29'). [finding] service-analytics read scope: compileScopedFilterToSql applies no whole-day upper bound and binds a temporal comparand as written, so an RLS $lte on a bare day drops the rest of that day in NativeSQL analytics #20733's caller-filter half.{ signed_at: { $between: ['2026-07-28', '2026-07-28'] } }: was no row, now 07-28.{today}upper bound is widened as the day it resolves to.Analytics
where, ObjectQL path: a bare-day$lteon adatetimemember reaches the engine as$ltthe next day, and the/analytics/sqlecho prints that half-open bound where it printed<=the named day. Rows unchanged.Draft preview:
$nullanswered (was refused 400). A row with no value now satisfies$ne,$ninand the negation of an equality when the comparand is the text"null"or"undefined"; the face compared those as text against the missing value.Cube face, measured on the published
MemoryAnalyticsService.queryat the base blob and at this head:where(fixture:disv1,v2, null, absent)find(){d: undefined}/{d: {$eq: undefined}}{d: {$ne: undefined}}{d: {$in: ['v1', undefined]}}{d: {$in: ['v1', null]}}{d: {$nin: ['v1', null]}}{d: {$gt: null}}{d: {$in: 'v1'}}{d: {$eq: {a: 1}}}{d: {$ne: {a: 1}}}{d: new Map()}{n: {$gt: 2n ** 60n}}{n: {$gt: 2n}}{d: {$between: ['v1', 'v2']}}{d: {$null: true}}{$or: [{d: 'v1'}, {n: 4}]}{d: {$ne: 'v1'}},{$not: {d: 'v1'}}(
find()here is the driver called directly, without the engine seam whose doors it relies on.) Every "refused 400" is the ADR-0112INVALID_FILTERenvelope every other analytics face already answers.Clause-②, measured
$or,$nulland$between(each refused at the base, each nowfind()'s rows), and the draft preview accepts$null.ANALYTICS_FILTER_CAPABILITIESnames$nulland$or.Hence
yes (narrowing).@objectstack/driver-memoryshipsminorwith the BREAKING banner, the migration and an ADR-0087not-required (no-migration-prescription)disposition;@objectstack/service-analyticsshipsminor(Clause-②: yes, widening only: the read scope and thewheredoor refuse nothing new).Mechanism assumptions, measured
normalizeAnalyticsFilterTreerather than insidenormalizeWhereComparands, because the array spelling reaches F10 throughparseFilterASTand never throughnormalizeWhereComparands, and becauselowerAnalyticsWhere's other readers want the authored condition. F11 lowers right after its ownnormalizeWhereComparandscall.wheredoor: tokens resolve upstream on every path that reaches it (AnalyticsService.resolveQueryTokensfromquery()andgenerateSql(), the per-request dataset-scope getter,DatasetExecutor.resolveSelectionTokensahead of the preview). Read scope: placeholder resolution is the compiler's first act, and the doors run aftercompileNodeon the scope as written (service-analytics: the NativeSQL read-scope compiler and the/analytics/sqlecho compile two scope shapes the shared comparand faces refuse (plain-object comparand under$eq, null member in$in): one scope, two answers across faces #20018), so the lowering sits between resolution and compilation. F5: no token resolution exists on that face.SecurityPluginand the real read-scope compiler on SQLite (a one-off run, not committed):getReadFilterreturned{"$and":[{"signed_at":{"$lt":"2026-07-29"}},{"due_on":{"$lte":"2026-07-28"}}]}, the read scope compiled("t"."signed_at" < ? AND "t"."due_on" <= ?)and answered c27 and c28, equal toSqlDriver.find. The committed pin holds the RLS half in both spellings the read scope can be handed (lowered by the RLS seam, and as written), so its answer no longer depends on which arrives.$andalone; F11 10 operators with$and/$or/$not. The lowering emits$and,$or,$lt,$gte,$lte,$null. Widened: F5$or+$null; F11$null. Nothing wider.timeDimensions).datecontrol) and [finding] driver-memory analytics: MemoryAnalyticsService runs neither shared comparand door, so a cube where {d: undefined} answers the null rows instead of a 400 #20734's table, plus a pin and an ablation per seam.placed_on: "0009-03-04"correctly, and…/queryreturns"1909-03-04"; adatetime0009-03-04T10:00Zreturns2004-09-03T10:00Z#20280 landed onmainas05a7547c9while this branch was open. Its datetime year floor lives in the engine's temporal comparand door (objectqltemporal-comparand-door.ts) and in@objectstack/core's temporal storage form; inpackages/spec/src/datait changed one comment. The two spec doors the cube face now runs are not changed by it. What the two share is the storage-form conversion both of the face's exits apply to a comparand (the driver'sfilterComparandStorageForm, which reads@objectstack/core): driver-sql on MySQL reads a year 0..99 back a century late — REST create storesplaced_on: "0009-03-04"correctly, and…/queryreturns"1909-03-04"; adatetime0009-03-04T10:00Zreturns2004-09-03T10:00Z#20280 changed that conversion, and this branch neither touches nor pre-empts it. Nothing in this PR's file surface changed onmain, so the branch is not re-merged; this PR's CI runs on the merge ref with driver-sql on MySQL reads a year 0..99 back a century late — REST create storesplaced_on: "0009-03-04"correctly, and…/queryreturns"1909-03-04"; adatetime0009-03-04T10:00Zreturns2004-09-03T10:00Z#20280 in it, and the merge queue adjudicates the rest.Evidence (head
92a449c2d,mainmerged atc90f9fb6e)@objectstack/service-analytics: 143 files, 3297 tests passed;typecheckgreen (baseline atbbe03f406: 141 files, 3267).@objectstack/driver-memory: 66 files, 1482 passed, 3 failed (the named gap above);typecheckgreen (baseline 65 files, 1470).node scripts/pm/dispatch-gates.mjs --commandsderived 63 families at this head; all 63 were run with their exit codes recorded, and--ranreports 63 derived, 63 run, 0 NOT-MEASURED, 0 UNRUN (a derived zero).check:dual-build-cjs-loadsandcheck:type-check-debtfirst answeredPREREQUISITE NOT METand were re-run green afterturbo run build --filter='./packages/*' --filter='./packages/*/*'. The six roster families under this card's directories (check-changeset-fixed,check:authz-resolver,check:error-code-casing,check:filter-alias-parity,check:object-def-param-keys,check:tenant-chokepoint) are green too.@objectstack/restanalytics-*(10 files, 150 tests),@objectstack/runtimeanalytics-*(3 files, 23),@objectstack/dogfoodanalytics-adhoc-query-isolation(24): green.eslint.config.mjsblockfiles: ['**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}'];eslint --no-inline-config --format jsonover the 38 changed source files reports 38 files, 0 errors, 0 warnings at92a449c2d; the config enables no type-aware linting (everyparserOptionsisecmaVersion/sourceType), so this diff cannot move a verdict on an untouched file.Acceptance notes
$ne,$nin,$notContainsand a$notoperand carry the guard twice. The same rows; the deletion cards remove the inner copy and update these pins.datetime; thewheredoor with nodeclaredFieldTypehook, and the read scope with nodeclaredValueShape, read none. Once a face's own bound copy is deleted, that path gets no whole-day bound unless its card gives it a typed reader.$notas a column constraint. No face meets that after this change (the analytics door spells it dotted; the engine, the read scope and the preview refuse the spelling; the cube face refuses$not). Noted for the module's owner; carrier: none.Patch round 1 (appended by the
domain:servicesseat)The named gap is closed. The claim's surface gained
packages/drivers/driver-memory/src/memory-driver-filter-logic-conformance.test.ts(amendment5911719808), andedb3e2de4applies the prepared 15-line patch to it and to nothing else:$notcase must still refuse; the$orcases moved to the answered column, which the file's case-by-case invariant holds tofind()'s rows;$nulljoins the declared-operator probe roster;$notContainspipeline dump shows the lowering's NULL escape.Evidence on head
edb3e2de4:@objectstack/driver-memory: 66 files, 1485 passed (the three pins green, nothing else moved);typecheckgreen.@objectstack/service-analytics: 143 files, 3297 passed.--ranreports 63 derived, 63 run, 0 NOT-MEASURED, 0 UNRUN; the six roster families green.mainwas not re-merged: none of the five commits afterc90f9fb6etouches this PR's files.Generated by Claude Code