fix(service-analytics)!: a cube or dataset dimension on a structured-JSON field is refused INVALID_FIELD / 400 at the analytics door, before any SQL is built (#20807) - #20886
Conversation
… refused at the analytics door (red) Pins first, before the fix. On the base they are red: - POST /api/v1/analytics/query with a cube or dataset dimension on a json field answers 200 with one group per serialized document on SQLite and 500 DATABASE_ERROR on PostgreSQL 16; /analytics/sql builds the statement. - POST /api/v1/analytics/dataset/query answers the same two ways. - The service door pins (both strategy faces, every structured-JSON type, the member as written, the one-predicate GUARD) are red; their controls are green. Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
…ON field at the analytics door A `dimensions` entry, or a bucketed `timeDimensions` entry, whose column is a declared structured-JSON field (the spec's STRUCTURED_JSON_TYPES, the class the engine's groupBy door reads) is refused INVALID_FIELD / 400 in `ensureCube`, naming the member the caller wrote, before either strategy builds anything. NativeSQLStrategy compiled GROUP BY by hand and never reached the engine's door: one group per serialized document on SQLite, 500 on PostgreSQL. The ObjectQL face reached the engine's door but was refused under `groupBy[0]`, a name the caller never wrote. Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
…clared join names; changeset A dataset dimension over an included relationship (`account.hq`) reached NativeSQLStrategy unjudged: measured, SQLite 200 with one group per document and PostgreSQL 500, like a base-object one. The door now reads the column the way the strategy compiles it: a bare identifier on the cube's object, a dotted identifier path on the object the cube's declared join for that path names. A path with no declared join is a synthetic traversal and stays unjudged. Pins: the service door and the dataset door gain the joined case and its text control. Changeset: service-analytics minor, BREAKING, Clause-② no (narrowing), ADR-0087 not-required (no-migration-prescription). Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
📓 Docs Drift CheckThis PR changes 1 package(s): ⛔ 3 release-owned page(s) name something this change touched. These are read-only:
What this run could not see
Coarse fallback — 10 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 9c6bd41c461f30fa13439c79ba0e3cfa0e7c093b && git checkout 9c6bd41c461f30fa13439c79ba0e3cfa0e7c093b
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 0803a8b871c81892e65cbe6978a282f927787155 075a46340e2cfc9dd57804c26318dabbfe2d366e && git checkout -B drift-repro 0803a8b871c81892e65cbe6978a282f927787155 && git merge --no-ff 075a46340e2cfc9dd57804c26318dabbfe2d366e
node scripts/docs-audit/affected-docs.mjs --json 0803a8b871c81892e65cbe6978a282f927787155
|
Contract reviewServed-tier: Inputs: card #20807 (body and all five comments: triage 5908875877, serial wait 5909886819, claim 5913045113, os-dev-report 5914264898, ACCEPT 5914351379), PR #20886 (body, six-file list, the net diff against ① Derived judgmentsThe diff implies one accept-set change and no public-surface change. Each judgment, named:
② Semver level
③ Boundary flagsEvery dev flag (
Implemented-by: VERDICT: PASS Generated by Claude Code |
Fixes #20807
Clause-②: no (narrowing)
What this changes
A dimension that GROUPS an analytics query, and whose column is a declared structured-JSON field (
json,composite,repeater,record,location,address,vector), is now refusedINVALID_FIELD/ 400 at the analytics door, naming the member the caller wrote, before either strategy builds anything. "Groups" means adimensionsentry, or atimeDimensionsentry that carries agranularity. It holds onPOST /api/v1/analytics/query, its dry runPOST /api/v1/analytics/sql, andPOST /api/v1/analytics/dataset/query, on every driver.The words, as
POST /api/v1/analytics/queryreturns them for a cube dimension:For a dataset dimension over an included relationship, the same words say
groups by field 'account.hq', whose column 'hq' the joined object 'X' declares as json. The thrown error isinvalidMemberError's envelope (code: 'INVALID_FIELD',status: 400,member,param,cube) plusfieldandobject.Landing site:
packages/services/service-analyticsonly.src/structured-json-dimension-door.ts(new):assertNoStructuredJsonDimension. The class is@objectstack/spec/data'sSTRUCTURED_JSON_TYPES, called, never re-listed. It is the same predicate the engine'sgroupBydoor (packages/objectql/src/group-by-structured-json-door.ts) reads, so there is one "is this field structured JSON" test for both doors. That file is read, not edited.src/analytics-service.ts: one private method,assertDimensionsGroupScalarColumns, called inensureCuberight after the dimension source-field gate on each of its three paths (inferred cube, augmented cube, declared cube). It supplies the two answers only the service has: the dimensionsqla member resolves to (declaredMemberEntry, the strategies' own'dimension'lookup, and the member itself when the cube declares none), and the column's declared type (sourceFieldMeta).NativeSQLStrategycompiles it. A bare identifier is a column of the cube's object. A dotted identifier path (account.hq) is its last segment, on the object the cube's DECLARED join for that path names, which is the alias the dataset compiler registers and the strategy joins. A path with no declared join is a synthetic traversal and is not judged.Before, measured on
origin/main793fb839Through the real
dispatcher-pluginroute over the serviceAnalyticsServicePlugincomposes on a realObjectQLengine, with both of its auto-bridges live (executeRawSqltoengine.execute,executeAggregatetoengine.aggregate). Three rows:titlex, x, y, and a differentmetadocument per row. Drivers:SqlDriveron SQLite (better-sqlite3), and on a private PostgreSQL 16.13 started for this run./analytics/query,dimensions: ['title'](text, the control)x2 ·y1/analytics/query, cube dimensionmeta(json)count1 each)DATABASE_ERROR(42883, "could not identify an equality operator for type json")/analytics/query, dataset dimensionmeta_doc(overmeta)DATABASE_ERROR/analytics/sql, cube dimensionmetaSELECT meta AS "meta", COUNT(*) AS "count" FROM ... GROUP BY meta/analytics/dataset/query, inline dataset dimensionmeta_docDATABASE_ERRORCounted at the engine for the json dimension: raw SQL 1,
engine.aggregate0, on both dialects, soNativeSQLStrategyanswered and the engine'sgroupBydoor never saw the query.A dataset dimension over a JOINED object's json field (
include: ['account'],field: 'account.hq') answered the same two ways: SQLite 200 with one group per document (2 groups), PostgreSQL 500 (42883). This was measured through the service on the first fix commitb1befe2a6, which judged only bare columns, and through/analytics/dataset/queryunder the ablation below. The second fix commit closes it.After, on this branch (
075a46340)Every refused row above answers
400 INVALID_FIELDnaming the member (meta,meta_doc,acct_hq), with zero raw-SQL statements and zero engine aggregates for the object. Thetitle,title_dimandacct_namecontrols answerx2 ·y1 (andA2 ·B1) from the native strategy, unchanged.Mechanism assumptions (zone 2): which held
793fb839as the red pins. See the table above and the raw-SQL / aggregate counts.dataset-compiler.ts:dimensions[d.name] = { sql: d.field }).DatasetExecutorpassesselection.dimensionsthrough as the cube query'sdimensions, so the member reachingensureCubeIS the name the selection wrote. The compiler checks only the relationship path (assertDeclared).STRUCTURED_JSON_TYPES, read at the engine door. The engine door's exported function takesgroupByentries and namesgroupBy[i], and@objectstack/objectqlis only a devDependency of this package, so the constant is what is called. No edit topackages/objectqlorpackages/spec.cube-registry.tsstores cubes and knows no field type.dataset-compiler.tsknows a dataset dimension's name and the declared type, but it compiles the whole dataset, whether or not a dimension is selected (a refusal there would refuse every selection of the dataset), and it does not see cube queries.analytics-service.ts'sensureCubeis the first step every door passes through that knows both the member as written and the column's declared type, ahead of strategy selection. The GUARD and the per-face unit cases show one answer on both strategies. Before this, the ObjectQL face reached the engine door but was refused undergroupBy[0].Temporal Conformance (live PG + MySQL)job setsOS_TEST_POSTGRES_URLfor three steps only:driver-sql,metadata-protocol'slive-postgresfiles, andruntime's cascade-delete matrix. It runsservice-analyticswithout a URL, and no step runs@objectstack/rest's or@objectstack/runtime's analytics pins. The PostgreSQL cells therefore sit beside each HTTP pin as a named skip without the URL, likepackages/rest/src/data-group-by-json-door.test.ts. They are red-capable and un-run in CI. The local PostgreSQL 16.13 runs are quoted below.no (narrowing). No new key reaches a published payload: the error envelope's members are the onesinvalidMemberErrorand the dimension source-field gate already attach. The accept set narrows: SQLite answered 200, and it now refuses. Changesetminor, BREAKING banner,Clause-②: no (narrowing). The ADR-0087 category measured isnot-required (no-migration-prescription): the package publishes, no ADR-0087 id covers a grouping target, and nothing authorable, exported or stored moves.Where the
/api/v1/analytics/querypin lives, measured. That route is served by@objectstack/runtime'sdispatcher-plugin(throughdomains/analytics.ts), not by@objectstack/rest. The REST package serves/analytics/dataset/query, andruntimedepends onrest, so a REST-package test cannot reach the runtime route. The cube-face pin is therefore a new file beside the repo's other/analytics/queryHTTP pins (packages/runtime/src/analytics-*.test.ts). The dataset-door pin is a new file inpackages/rest/src/. Both are new pin files only.Tests
All at
075a46340, the final commit.packages/services/service-analytics/src/__tests__/dimension-structured-json-door.test.ts: 13 passed. It covers every structured-JSON type on both strategy faces (native and ObjectQL), with the envelope (code,status,member,param,field,object) and zero raw SQL and zero aggregates. It also covers a cube key over another column, the cube-qualified spelling, a bucketed time dimension on both faces, a declared default granularity, a dataset dimension, a dataset dimension over an included relationship (judged on the joined object), an ad-hoc inferred cube, and the dry run. Controls: a text dimension is served, an unknown field keeps the existence gate's answer first, a synthetic dotted traversal is not judged, and a host withoutsourceFieldMetastands down. GUARD: over everyFieldType, the refused types are exactlySTRUCTURED_JSON_TYPES.packages/runtime/src/analytics-json-dimension-door.test.ts(POST /api/v1/analytics/queryand/sql): 8 passed, SQLite 4 and live PostgreSQL 16.13 4.packages/rest/src/analytics-dataset-json-dimension-door.test.ts(POST /api/v1/analytics/dataset/query): 6 passed, SQLite 3 and live PostgreSQL 16.13 3.46ee85eda(the pins on the base code) the unit file was 8 failed and 3 passed, the runtime file 6 failed and 2 passed, and the REST file 2 failed and 2 passed. The failures were the refusals; the controls were green.pnpm --filter @objectstack/service-analytics exec vitest run: 145 files / 3325 passed.typecheck, exit 0:@objectstack/service-analytics(tsc --listFilesOnlyincludes the new door and the new test);@objectstack/rest(the new test is in its test program, andcheck:test-typecheckholds at 0 files / 0 errors);@objectstack/runtime(check:test-typecheckholds at 27 files / 190 errors / 68 signatures, unchanged).exportsentry of@objectstack/service-analyticschanges, so only behaviour moves. A census ofexamples/at793fb839found no producer grouping by a structured-JSON field: 5 files declare a cube or dataset, 18 distinct dimension sources, none of them structured JSON.Reverse verification (ablation), from the committed fix at
075a46340. It ran throughscripts/ablation-replace.mjsin WRAP mode, trap-restored, under the verify lock. The door's own verdict line gained an always-truecontinueguard keyed on the marker__ablated_20807__. On disk: anchor 1 to 0, blob47ce2b2acc11toa286ca8a75ae.service-analyticswas rebuilt, andablation-dist-preflightfound the marker in 2 built files (dist/index.js,dist/index.cjs)./analytics/querypin: 6 failed / 2 passed. On SQLite the cube and dataset dimensions answered 200 with 3 groups and the dry run served the statement. On PostgreSQL they answered 500DATABASE_ERROR. The controls stayed green./analytics/dataset/querypin: 4 failed / 2 passed.meta_docandacct_hqanswered SQLite 200 and PostgreSQL 500. The controls stayed green.47ce2b2acc11),git diff HEADis empty, and the whole-treegit status --porcelainis empty. After a rebuild,--absentfound the marker absent from all 6 built files and the tree clean. The re-run was green: 13, 8 and 6 passed.b1befe2a6, before the joined-column extension, read 8/3, 6/2 and 2/2, the same direction.Gates
node scripts/pm/dispatch-gates.mjs --commands(no paths) at075a46340derived 62 commands over the 6 changed paths. That is the same list the PM derived at793fb839. The four roster families the order names were run too:node scripts/check-changeset-fixed.mjs,pnpm check:authz-resolver,pnpm check:error-code-casingandpnpm check:filter-alias-parity.--ranreconciles: 62 derived, 62 run, 0 NOT-MEASURED, 0 UNRUN. All 66 commands exit 0.check:dual-build-cjs-loadsandcheck:type-check-debtfirst exited 3 (PREREQUISITE NOT MET) and were re-run to exit 0 after a fullturbo run build --filter='./packages/*' --filter='./packages/*/*'. Among them:check:adr-0087-registration --base origin/main:[BREAKING+bang+clause-②-narrowing] not-required (no-migration-prescription)accepted.check:changeset-no-major,check:empty-changeset,check:doc-authoring,check:nul-bytes,check:issue-citations.check:cross-package-test-inputs,check:test-source-alias,check:driver-memory-census,check:rest-log-spy-declared,check:engine-double-contract,check:query-options-erasure,check:type-check-coverage,check:type-check-debt(re-measure: none above its record).Note:
dispatch-gatesflagged the tree as 6 commits behindorigin/main2d5fe76f4. None of those commits touches this diff's paths,service-analytics, the analytics routes, the engine door orSTRUCTURED_JSON_TYPES, somainwas not merged (the order merges only on a surface hit). The merge ref CI builds covers the joint tree.Lint, narrowed and proven:
pnpm exec eslint --no-inline-config --format jsonover the 5 changed.tsfiles at075a46340found 5 files, 0 errors, 0 warnings. Three facts make this narrowing a measurement:isPathIgnoredanswersfalsefor all 5.parserOptions.projectandprojectServicearenullfor every file, so type-aware linting is not enabled.Changeset
.changeset/20807-analytics-json-dimension-refused.md:@objectstack/service-analyticsminor, BREAKING banner,Clause-②: no (narrowing), and exactly one ADR-0087 marker,not-required (no-migration-prescription), in the form the engine door's changeset uses. It states the refused shape, names the refusal's code, and says who is affected and what is unchanged. No export or published type changes.Acceptance notes
POST /api/v1/analytics/queryon PostgreSQL 16, thetextcontrol's rows came back{"title":"x","count":"2"}whilefieldssaid{"name":"count","type":"number"}. SQLite returned2. The registered dataset'srow_countbehaved the same. This is the result-typing class triage directed out of this card.os validate(the built CLI at075a46340) passes a stack whose dataset declares a dimension over ajsonfield: exit 0, "Validation passed". The runtime now refuses that dimension at query time. The same stack with a measureavgover that field is refused bymeasure-aggregate-field-type-refused, so the command does judge dataset members against declared types.timeDimensionsentry with nogranularityover a json field. It bounds a range and groups nothing, so it is a filter's question, not this door's.sourceFieldMeta.queryDatasetwithpreviewDraftsover a pending seed evaluates in memory (evaluateAnalyticsQueryOverRows) and does not passensureCube.MemoryAnalyticsService(driver-memory's cube face) is [finding] driver-memory analytics: MemoryAnalyticsService drops an array (FilterArray)whereand answers every row, where the object spelling filters and the engine refuses 400 #20859's position and is not touched.INVALID_FIELD("does not have") answer, because the dimension source-field gate runs first.where/ preview door, the read scope) and the memory cube face's door, with the F5 / F11 output vocabulary #20810 is not addressed here: it lowers filters, and this card refuses dimensions.Generated by Claude Code