fix(service-analytics)!: the nested-relation filter gets the engine's answer on every analytics face — the related object read as the caller, capped (#20887) - #20916
Conversation
…lytics faces to the engine's answer (red)
The cube read (both strategies), the dataset door, the SQL echo and a read
scope carrying `{ relation: { field: value } }` are pinned against the
engine's own answer over one fixture with the real security layer: the
engine's rows for single- and multi-valued relations, the related row
scope, $not and $or; its 403 for a related field the caller cannot read;
its 400 past the cap and at an aggregation's filter. Plus the analytics
seams: the native strategy declines the form from every producer it
compiles, the ObjectQL strategy hands the engine the form as written, and
the read-scope compiler's refusal names the route that serves it.
Red on the base: 10 of 10 route pins, 7 of 9 unit pins (the 2 controls
pass).
Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H
Co-authored-by: Claude <noreply@anthropic.com>
…engine on every analytics face
`{ relation: { field: value } }` is the engine's form (served in `where`
at the #5930 seam: the related object read as the caller, capped). The
analytics door used to flatten it to a dotted member: the native strategy
LEFT JOINed the related table (its row scope applied, its field
permissions and the cap did not, a multi-valued relation and an undeclared
join missed), and the ObjectQL strategy refused it as a cross-object
filter.
- filter-normalizer: a nested-relation condition becomes a `relation`
node carried as written (no dotted flattening, no local NULL guard);
`findNestedRelationCondition` is the routing detector.
- NativeSQLStrategy.canHandle declines a query whose where, dataset
filter, requested measure filter, or base/joined read scope carries the
form (the mechanism of the cross-field decline); its compiler refuses a
relation node bare, as routing drift.
- ObjectQLStrategy hands the engine the condition as written, as its own
conjunct; the display-SQL echo declines it in the where-door envelope.
- read-scope-sql keeps its fail-closed refusal of the form (a synchronous
SQL compile that reads no other object), in words naming the engine
route; the empty/mixed value object keeps its old words.
- The no-strategy error names the nested-relation decline.
Pins that recorded the flattening are re-judged (respelled to the dotted
cube member where they were about the traversal, replaced where they
pinned the removed branch).
Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H
Co-authored-by: Claude <noreply@anthropic.com>
… shared lowering
Under $not the shared lowering read the relation key as a column of the
queried object and guarded it; the engine hand-off spells that guard
`$ne: null`, which the SQL driver refuses over a multi-valued relation's
JSON column, so `{ $not: { owners: { region: 'NA' } } }` answered
INVALID_FILTER where the engine answers its rows. Each nested-relation
condition now stands in the lowered condition as a total sentinel and is
restored as the relation node; the engine guards what it lowers the
condition to itself.
Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H
Co-authored-by: Claude <noreply@anthropic.com>
…ine answer Clause-② measured as yes (narrowing): served where the analytics faces refused or failed, refused where the native join answered without the caller's field permissions or the cap. Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
…m keeps its declared refusal on the native path B4 measured: the read scope's SQL compile is a synchronous string builder holding the caller's context for placeholders and no data engine, so it cannot read the related object as the caller. The native strategy no longer declines on a read scope carrying the form: routing it away traded the declared READ_SCOPE_COMPILE_FAILED (policy withheld) for whatever the next strategy answered, and a generic fault on a host without one. The compile keeps its fail-closed refusal, in words that name the engine route; the engine-aggregate path still hands the scope to the engine, which serves it as the caller. Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
Brings the gate derivation onto a fresh tree (a derived-from gate script changed on main); no commit on main touched this branch's surface. Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
📓 Docs Drift CheckThis PR changes 1 package(s): 3 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:
⛔ 2 release-owned page(s) also name something this change touched. These are read-only:
What this run could not see
Coarse fallback — 10 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin cfc9af67dce65396e7d9bbff98aa8a49c093ba48 && git checkout cfc9af67dce65396e7d9bbff98aa8a49c093ba48
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 212d613ca03d2dd3de7147ebd0eb759667e2ffa9 6b6bffb3eaacd348a491cc185fcaf506f38c0106 && git checkout -B drift-repro 212d613ca03d2dd3de7147ebd0eb759667e2ffa9 && git merge --no-ff 6b6bffb3eaacd348a491cc185fcaf506f38c0106
node scripts/docs-audit/affected-docs.mjs --json 212d613ca03d2dd3de7147ebd0eb759667e2ffa9
|
main's 975b248 touched packages/services/service-analytics (the dataset compiler and one of its pins), this card's surface, so it is merged before patch round 2. Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
…census The rest pin packages/rest/src/analytics-nested-relation-filter.test.ts calls the real AnalyticsService (the cube read) five times, to compare its answer for a nested-relation filter with the engine's. Those are producer reads, not SDK callers: one NOT_SDK ledger row, with the service-receiver count and the NOT_SDK total moved from 1 to 6, and the two-file set asserted. Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
…nput The census ledger row added for the rest pin names packages/rest/src/analytics-nested-relation-filter.test.ts: it pins that file's five producer reads of analytics.query. That makes it a real input of @objectstack/client's tests, so it is declared per-file in the client's cross-package input globs and mirrored into client#test's turbo inputs. A changed call count in the rest pin now re-runs the client suite. Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
Contract reviewServed-tier: PR #20916 (card #20887), reviewed at 2026-09-30T19:46Z by an isolated contract-review subagent of the ① Derived judgmentsPublic surface
Accept set, per face (the ruling's line: one answer on every face, the engine's)
② Semver level
③ Boundary flagsEvery dev flag and every
Implemented-by: VERDICT: PASS |
main's #20931 (the field-read admission gate), #20955 (the queryable-field gate), #20954 (plugin-security's comparand guard) and #20962 (relationship path objects in the admitted and scoped set) touched packages/services/service-analytics. The merge is clean at the text level; both sides' additions to analytics-service.ts and native-sql-strategy.ts are kept whole. Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
Contract reviewServed-tier: Delta review of PR #20916 (card #20887) on the merge head, rendered at 2026-10-01T00:38Z by an isolated contract-review subagent of the ① Derived judgmentsThe merge,
The interaction with what
Docs
② Semver level
③ Boundary flagsRound 4's report (
Implemented-by: VERDICT: PASS |
|
Heads-up from PR #20979 (#20897) edits the same three places this PR does: |
Fixes #20887
Clause-②: yes (narrowing)
The analytics half of ruling 5907789183, whose parent card is #20802 (its engine half landed as #20872,
ca5408c62). The nested-relation filter{ relation: { field: value } }now gets ONE answer on every analytics face, and it is the engine's: the related object read as the caller (its row scope and field permissions), capped atRELATION_FILTER_ID_CAP, a multi-valued relation matching on any member. The analytics layer holds no copy of that rule. The native-SQL strategy declines a query carrying the form, and the engine-aggregate strategy hands the form to the engine as written.Per face
The engine's answer for the same filter, computed in the same test over the same rows, is the reference for every cell. Fixture: a ledger with
owner(lookup) andowners(multiple lookup) to an owner object; the member cannot readowner.secret, and its row scope hides owners in regionHIDDEN. Past the cap means 1,001 matching owners. Measured with the realSecurityPlugin,ObjectQLandSqlDriver(SQLite).POST /api/v1/analytics/query(AnalyticsService.query)$not,$or(was 500DATABASE_ERROR: the join named a tableownerthat does not exist)PERMISSION_DENIED, as the engine (was 500)INVALID_FILTER, as the engine (was 500)INVALID_FIELD, "cannot evaluate a cross-object filter")POST /api/v1/analytics/dataset/query, the datasetincludesownerDATASET_INVALID; under$notthe member gotbwhere the engine answersb, d)filtercarrying the formINVALID_FILTER, as the engine refuses it at an aggregation'sfilter(was: native counted it through the JOIN; ObjectQL 400INVALID_FIELD)POST /api/v1/analytics/sqlINVALID_FILTER, naming the served route (was: native printed the JOIN; ObjectQL 400)getReadScope)READ_SCOPE_COMPILE_FAILED, policy withheld, words now naming the route (outcome unchanged)Mechanism assumptions, measured
{ owner: { region: 'NA' } }is d1, d3; the multi-valued form is d1, d3;{ owner: { secret: 's1' } }is 403PERMISSION_DENIEDnamingsecret(a system caller gets d1, d3); regionHIDDENgives no rows (a system caller gets d4); past the cap is 400INVALID_FILTERfor both spellings;$notgives d2, d4; the$orgives d1, d2, d3;{ owner: {} }and a second level are 400.RELATION_FILTER_ID_CAPis exported (packages/objectql/src/index.ts:148), and nothing here imports it: the analytics layer never counts ids, the engine does.WHEREconjunct, the field permissions did not, nothing bounded the match, and a multi-valued relation or an undeclared join failed. The ObjectQL path refused the form outright.@objectstack/objectqlexports only the cap. The lowering (admitRelationCondition,lowerRelationSite) is module-internal, and this package has@objectstack/objectqlas a dev dependency only. The route that needs no export: the engine-aggregate strategy hands the condition toengine.aggregatethroughexecuteAggregate, with the caller's context. No export was needed, and there is no second permission rule.compileScopedFilterToSqlis a synchronous string builder. It holds the caller'sExecutionContextfor placeholders only, and no data engine, so its compile cannot run the inner read as the caller. Routing a read scope carrying the form to the engine instead was built and measured, then withdrawn: on a native-only host it traded the declaredREAD_SCOPE_COMPILE_FAILED(policy withheld) for a generic no-strategy fault (packages/rest/src/analytics-read-scope-refusal-envelope.test.tswent red). No in-repo producer emits the form in a scope: the RLS compiler refuses a relation traversal when it compiles the policy. On the ObjectQL path the scope reaches the engine as before, and the engine serves it as the caller.yes (narrowing). Widening: the cube read (both strategies), the ObjectQL dataset door, a multi-valued relation and a dataset without the declared join on the native path, and a dataset's ownfilteron the ObjectQL path all now serve the form (they answered 500 or 400). Narrowing: on the native path the dataset door now refuses a condition on a related field the caller cannot read (was rows), a match past the cap (was an empty 200), and a measure filter carrying the form (was a count). The SQL echo refuses the form. And a query combining the form with something only the native strategy serves (a cross-object measure, a multi-hop dimension) is refused by the engine-aggregate path.@objectstack/service-analyticsshipsminorwith the BREAKING banner and an ADR-0087not-required (no-migration-prescription)disposition;check-adr-0087-registrationandcheck-changeset-no-majorpass.content/docs/**page states how the analytics read or the read scope treats the nested form.data-engine.mdx, andquery-syntax.mdx(docs(query-syntax): Filtering Across Relationships states the served nested-relation form #20906, which landed during this work), describe the engine only.What changed
strategies/filter-normalizer.ts: a nested-relation condition becomes arelationnode carrying the condition as written. It is no longer flattened to the dotted member.shieldNestedRelationsholds it out of the shared lowering, because under$notthe lowering guarded the relation column, and this package's engine hand-off spells that guard$ne: null, whichdriver-sqlrefuses over a multi-valued JSON column. Measured: the multi-valued$notpin went red before the shield, and the engine guards what it lowers the condition to itself.findNestedRelationConditionis the routing detector.strategies/native-sql-strategy.ts:canHandledeclines when thewhere, the dataset's ownfilteror a requested measure'sfiltercarries the form. This is the mechanism of the cross-field decline (maintainer ruling 2026-08-12, Q1 = B). Its compiler refuses arelationnode bare, as routing drift.strategies/objectql-strategy.ts: the condition goes to the engine as its own conjunct, under the key the author wrote. The display-SQL echo declines it.read-scope-sql.ts: the nested-relation form's refusal has its own words, naming the route. An empty or mixed value object keeps the old words.analytics-service.ts: the no-strategy error names the nested-relation decline.Pins, red first (
568727629)packages/rest/src/analytics-nested-relation-filter.test.ts: both compositions, the cube read and the dataset door through its route, against the engine's answer. It was red 10 of 10 on the base, and is 10 of 10 green now.packages/services/service-analytics/src/__tests__/nested-relation-engine-handoff.test.ts: the native decline per producer, the ObjectQL hand-off as written, the compile backstop and the read-scope words. It was 7 red with 2 controls green on the base, and is 9 of 9 green now.Ablations, predicted before running, at
5bb764181Each ablation mutated the committed file through
scripts/ablation-replace.mjs(anchor hit once, blob moved), rebuilt@objectstack/service-analytics, and passedablation-dist-preflight(the marker present in 2 built files). It then ran both pin files, pluswhere-door-shared-lowering-seam.test.tsin the unit run. The restore leg proved the blob equal to HEAD andgit diff HEADempty, rebuilt, and found the marker absent from all 6 built files. Every observed count equals its prediction.d(a hidden owner's row), and the unreadable field answered rows$notA first round at
dca1af7cbmatched its own predictions too, including A5, the read-scope decline arm, which B4's correction removed from the code.Pins re-judged
These pins recorded the flattening this change removes, so each was re-judged:
filter-normalizer-not-null-safe,icontains-text-comparand-refusal;relationnode where the pin was about acceptance:where-equality-slot-list-refusal,where-face-arms-refusal,where-type-face-refusal,filter-normalizer-mixed-wrapper's pure-shape block;guardFieldEntryrecursion row (the engine refuses that inner wrapper,INVALID_FILTER/ 400, measured),where-door-shared-lowering-seam,infer-cube-relation-traversal,infer-cube-where-spelling-parity, andwhere-source-field-gate, which now judges the relation fieldowneras a column of the queried object;read-scope-sql,read-scope-not-null-safe,read-scope-undefined-comparand, andread-scope-refusal-envelope, which gains row 17 because the nested-relation form now has a throw site of its own.Verification
@objectstack/service-analytics:test146 files, 3334 passed;typecheckexit 0, with 146 of 146 test files in the tsc program (--listFiles). Both at4d383dac0, after mergingmain.@objectstack/rest: the fulllocalproject, 239 files, 4662 passed and 106 skipped, at5bb764181. The merge brought no rest or analytics change. At4d383dac0, the new pin, the read-scope envelope pin and the engine half's permission pin: 3 files, 21 passed.typecheckpasses, including the test layer (check:test-typecheckOK).@objectstack/runtimeanalytics-*pluscross-field-refusal-operand-withhold, 5 files, 38 passed and 4 skipped;@objectstack/clientanalytics-automation-json-erasure, 7 passed.4d383dac0:dispatch-gates --commandsderived 62. All 62 were run, plus the 4 roster families (check-changeset-fixed,check:authz-resolver,check:error-code-casing,check:filter-alias-parity), all exit 0.dispatch-gates --ran: 62 derived, 62 run, 0 NOT-MEASURED, 0 UNRUN.check:dual-build-cjs-loadsandcheck:type-check-debtfirst exited 3 (prerequisite not met) and were re-run green afterturbo run buildover./packages/*.4d383dac0. The population is the 21 changed.tsfiles, none ignored by eslint's own config (isPathIgnoredfalse for all 21).eslint --no-inline-config --format jsonover them gives 21 files, 0 errors, 0 warnings.parserOptions.projectandprojectServiceare unset for all 21, so no type-aware lint runs and no untouched file's verdict can move.data-nested-object-door.test.ts; this file's axis is the analytics faces), the dogfood and integration lanes, and the whole-workspace typecheck. All are left to CI.Acceptance notes
where.$and[0].ownerwhere the author wrotewhere.owner.packages/types/src/error-leak.test.tskeeps a hand-written stand-in of the read-scope refusal shapes. Its nested-relation line is the old wording. It is a heuristic fixture, not a pin of this module, and it stays green.MemoryAnalyticsService(driver-memory's cube face, [finding] driver-memory analytics: MemoryAnalyticsService drops an array (FilterArray)whereand answers every row, where the object spelling filters and the engine refuses 400 #20859's position) is not touched, and its answer for the form is not measured here. The draft preview refuses the form as an operator it cannot evaluate, unchanged.Patch rounds (the seat's append from the dev's reports
5917211340,5917807320and5918233769; the dev writes a body only once)Patch round 1
Test Core (3/6)went red on4d383dac0, inpackages/client'senvelope-caller-census.test.ts: 2 of its tests failed. Reproduced locally: the client suite fails 1 file and 2 tests at4d383dac0, and passes 50 files and 641 tests at the merge base9509ea106.Root cause. The census walks the whole workspace for call sites of
analytics.query(and requires a hand-ledger row for each one. This PR's new pin,packages/rest/src/analytics-nested-relation-filter.test.ts, calls the realAnalyticsService'sanalytics.queryfive times. Those are producer reads, the census'sNOT_SDKclass, and the ledger has no row for them. The failing assertions are the §3 key comparison (the one extra key is that file,analytics.query,service, 5) and the §2 producer-receiver count (1 expected, 6 found).What it is not. It is not a product defect. It is not a client pin of the nested-relation form or of the read-scope wording either.
The fix, pending the seat. It lives in
packages/client/src/envelope-caller-census.test.ts, which is outside this card's claim surface. It adds oneNOT_SDKledger row with a count of 5, and moves the two producer-read counts from 1 to 6. Measured on a scratch copy of that file: 20 of 20 tests passed. The copy was restored byte-identical, and nothing was committed.Patch round 2
The seat authorised the census remedy, round 1's option A, for one file:
packages/client/src/envelope-caller-census.test.ts.975b2481c([finding] two more JSON-stored columns as a group or distinct key answer 500 on PostgreSQL:groupByon amultiple: trueselect, andcount_distincton ajsonfield #20808) touchedpackages/services/service-analytics, soorigin/mainwas merged into the branch as1fdaff7e5(no rebase). The merge was clean, with no regeneration pending.7eb2ecf20.NOT_SDKledger row forpackages/rest/src/analytics-nested-relation-filter.test.ts(analytics.query,service, count 5).verdictTotal('NOT_SDK')goes from 1 to 6, and its test title changes with it.7eb2ecf20. Each exit code was captured before any pipe.pnpm --filter @objectstack/client test: exit 0, 50 files and 641 tests passed. The census file run alone passed 20 of 20.pnpm --filter @objectstack/client typecheck: exit 0.tsc --noEmitpassed, andcheck:test-typecheckanswered OK.@objectstack/service-analyticstest: exit 0, 146 files and 3335 tests passed. Its typecheck: exit 0.analytics-nested-relation-filter,analytics-read-scope-refusal-envelope,data-nested-relation-permission): exit 0, 3 files and 21 tests passed..tsfiles: 0 errors and 0 warnings. The config ignores none of them and lints none type-aware, so this diff cannot move a verdict on an untouched file.check:cross-package-test-inputs.@objectstack/client's declared cross-package input globs do not cover it. The gate is green at1fdaff7e5, the commit before.scripts/cross-package-test-inputs.mjs, and mirror it inturbo.json's@objectstack/client#testinputs.check-ci-filter-parityboth exit 0. The two files were then restored byte-identical.Patch round 3
The seat authorised the gate's own remedy for
check:cross-package-test-inputs, in two files.975b2481ctouched this card's surface, the census or either of the two files, so there was no merge. The commits checked weredef279a39,4d0b9cd54andd78a0bda0.2881f478c.scripts/cross-package-test-inputs.mjs: in@objectstack/client's entry, one per-file glob,packages/rest/src/analytics-nested-relation-filter.test.ts, with a 3-line comment.turbo.json:$TURBO_ROOT$/packages/rest/src/analytics-nested-relation-filter.test.tsin@objectstack/client#test's inputs. The line before it gains the comma JSON requires.2881f478c.check-ci-filter-parity --self-testandcheck:select-shard-packages. All 93 commands exit 0.check:cross-package-test-inputs(with--self-test) is green: "OK: 29 package(s) read outside themselves, all declared".check-ci-filter-parityis green: "all 188 declared cross-package glob(s) (135 unique) are covered".check:turbo-task-graphis green.pnpm --filter @objectstack/client test, as the control: exit 0, 50 files and 641 tests passed, the same as at7eb2ecf20. The declaration moved no verdict.--union-into, given a diff of the rest pin alone, now pulls@objectstack/clientinto the run (8 packages). At7eb2ecf20it did not (7 packages). No other package changed.--dry=jsonbefore and after, over build, test, test:repo and typecheck (303 tasks):@objectstack/client#testmoved throughturbo.json: its task definition changed, and the rest pin is a new input.scripts/cross-package-test-inputs.mjs, an input they declare, changed. They arecli#test,plugin-auth#test,vitest-filter-preflight#test,objectql#test:repo,runtime#test:repoandspec#test:repo..tsand.mjsfiles: 0 errors and 0 warnings. The config ignores none of them and lints none type-aware.Patch round 4 (the seat's append from the dev's report
5922062971)mainwas merged (no rebase) to take in four landings inservice-analytics:plugin-security's comparand guard.The merge,
6b6bffb3e. It is clean at the text level, inanalytics-service.tsand innative-sql-strategy.ts. Every line either side added is present in the merged files, checked line by line.What the landed gate and object set do with the
relationnode. This was measured on the merged tree, in the shipped composition (the realSecurityPluginoverObjectQLon SQLite), under both strategies.collectFilterLeavesyields the nested form's relation field as its member:{ owner: { region: 'NA' } }givesowner, with operatorrelation. It does the same under$notand inside$or. So the field gate judges the relation field on the base object.owneris refused by the gate: 403PERMISSION_DENIED, in the engine's own words, with no engine call made.queryObjects. The security service is asked only about the base object.engine.find, and never answered:mainalone, even a readable nested condition was refused 403, "reading "owner" is not permitted", because the flattenedowner.regionnamed the relation field as an object to admit. With this branch, the answer is the engine's.Measured at
6b6bffb3e. Every run was under the shared lock, with each exit code captured before any pipe.@objectstack/service-analytics: tests exit 0 (149 files, 3434 tests), and typecheck exits 0.analytics-nested-relation-filter: 10analytics-read-scope-refusal-envelope: 8data-nested-relation-permission: 3analytics-field-permission-gate: 12analytics-relationship-path-admission: 28.tsand.mjsfiles: 0 errors and 0 warnings.Acceptance notes.
{ 'owner.region': 'NA' }is refused 403, "reading "owner" is not permitted". The hop object is taken from the alias, because an inferred cube declares no join. This is the same onorigin/main, and it is outside this card; it went to the seat as a finding.queryObjects, so a host-suppliedgetReadScopeis not asked about it. In the shipped composition that provider is the security service'sgetReadFilter, the same row scope the engine applies when it reads the related object as the caller. That case is pinned inanalytics-nested-relation-filter("the related row scope").Generated by Claude Code