Merge remote-tracking branch 'origin/main' into claude/issue-8262-reaper-verify-not-infer

Author: claude

.changeset/formula-filter-refusal-adr-0087-entry.md

@@ -0,0 +1,32 @@
+---
+"@objectstack/spec": patch
+---
+
+<!-- adr-0087: registered engine-find-formula-filter-refused -->
+
+docs(spec): register the FILTER-axis formula refusal in the ADR-0087 ledger (#8370)
+
+The refusal itself shipped in 17.0.0 (#8296 / PR #8369): a `where` naming a
+`formula` field is `400 INVALID_FIELD` at both doors — the REST ingress
+(`assertFilterFieldsExist`) and the engine's own filter seam
+(`assertFilterIsMaterializable`), which saved reports, flows and dashboard
+widgets reach directly. It shipped with **no** ADR-0087 semantic entry, so
+`objectstack migrate meta`, `spec-changes.json` and the generated upgrade guide
+said nothing about it.
+
+Its SORT-axis twin (#7095, `engine-find-formula-order-by-refused`) carries one,
+for the identical shape. This adds the FILTER-axis sibling —
+`engine-find-formula-filter-refused` under protocol 17 — and regenerates the two
+projections of the registry.
+
+For a code-path API there is no `sys_metadata` row for the D2 chain to rewrite
+and no mechanical rewrite in either direction (the platform cannot invent the
+stored column, and it must not filter post-hoc — `driver.find` has already
+applied `limit` / `offset`, so a post-hoc predicate would filter an arbitrary
+PAGE), which makes the ledger entry the only notification channel this class
+has. The remedy it prescribes is the one the sort and search axes already
+prescribe, in the same words: denormalise the value onto a stored field written
+when the source changes, and filter that. `summary` and `autonumber` fields need
+no action — both get real maintained columns and filter correctly.
+
+No behaviour changes: registration and regenerated artifacts only.

docs/protocol-upgrade-guide.md

@@ -410,6 +410,19 @@ Ruled 2026-08-12 on #7589 (Option B): a dotted entry the engine cannot resolve i
 
 This is a CODE-path API, not stored metadata, so — like `engine-find-formula-order-by-refused` at this step — there is no `sys_metadata` row for the D2 chain to rewrite and the ledger entry is the notification channel. No mechanical rewrite exists: the platform cannot decide between `expand` and denormalisation for the caller, and it must not resolve the path itself — no driver ever did, and inventing a join here is a feature decision, not a migration. #7589, #7532, #7601, #3821, #5918, ADR-0112.
   - Done when: No `engine.find` / `engine.findOne` call site passes a dotted `fields` entry, no flow `get_record` config authors one, and no saved report's `query.fields` names one — grep flow definitions and report definitions for a `fields` entry containing a `.`, and rewrite each to `expand` (keeping the reference column projected) or to a denormalised stored column. Reads complete with no `INVALID_FIELD` whose message says "follows the relationship" or "a dotted path", and no "Failed to expand relationship field" warning whose error text does.
+- **`engine-find-formula-filter-refused`** — `a `where` / filter naming a `formula` field — at BOTH doors: the REST ingress (`assertFilterFieldsExist`, covering everything that reaches `findData`) and the engine seam itself (`engine.find` / `findOne` / `count` / `aggregate` / `update` / `delete`), which saved reports, flows and dashboard widgets reach directly` → denormalise the value onto the object (a stored field, written when the source changes) and filter that — deliberately the same remedy, in the same words, the SORT axis prescribes (#6924 / #6994 / #7095) and the SEARCH axis has prescribed since #6674; `summary` and `autonumber` fields need NO action, because both get real maintained columns and filter correctly
+  - Why not automatic: `formula` is the one field type no driver materialises a column for, and FILTER was the last of the three query axes still fail-open on it: SORT refuses it (#6994 at the ingress, #7095 at the engine) and SEARCH refuses it by name (#6674), while a `where` on a `formula` field cleared every gate precisely BECAUSE the object declares the field, reached a driver with no column behind it, and answered 200 with zero rows. Measured on a real `ObjectQL` with `is_open` a `formula` over the stored `status` column: `where {is_open: true}` and `where {is_open: false}` each returned 0 rows with NO error, while the controls `where {status: 'open'}` returned 4 rows and `where {subtask_total: 5}` (a `summary`, which HAS a column) returned 1 row.
+
+BOTH directions are wrong and the `false` one is the dangerous one: the same predicate against a STORED boolean returns every matching row, so a filter meaning "not yet done" silently became "no records at all" — a row SET changed under a 200, which no amount of inspecting the response can reveal, and the formula READS correctly in that very same response, so the field is visibly populated and simultaneously unfilterable. That is strictly worse than the sort axis it mirrors: a refused sort returns the same rows in a different order, a refused filter changes which rows exist.
+
+Both doors now refuse it with `400 INVALID_FIELD` (#8296 / PR #8369), naming the offending key path and carrying the remedy sentence — the ingress gate (`assertFilterFieldsExist`, `@objectstack/metadata-protocol`) for everything reaching `findData`, and `assertFilterIsMaterializable` (`@objectstack/objectql`, `filter-comparand-shape.ts`) at the engine's own filter seam, which every caller-supplied `where` passes through whichever verb it arrived by. Both judge the field by the SAME `@objectstack/spec/data` predicate the SEARCH axis uses (`isVirtualSearchField` / `SEARCH_VIRTUAL_TYPES`, which holds `formula` and nothing else), so gate and drivers cannot disagree about which types have a column: a gate widened to the spec's `COMPUTED_VALUE_TYPES` (the WRITE contract) would refuse two working types. DOTTED filter paths are deliberately not judged on this axis at either door.
+
+This is a CODE-path API, not stored metadata, so — like `engine-find-formula-order-by-refused` and `engine-dotted-projection-refused` at this step — there is no `sys_metadata` row for the D2 chain to rewrite and this ledger entry is the notification channel. No mechanical rewrite exists in either direction: the platform cannot invent the stored column the remedy prescribes, and it must not filter post-hoc instead — `driver.find` has already applied `limit` / `offset`, so a predicate applied after the formulas are evaluated would filter an ARBITRARY PAGE, which looks correct on small result sets and is wrong the moment pagination is involved.
+
+AUTHOR-REACHABLE SURFACES are why this is not merely a code-side note. A saved report's `query.filter` (`sys_saved_report`) is forwarded VERBATIM into `engine.find` by `plugin-reports` (`report-service.ts`, `where: q.filter`), bypassing the ingress gate entirely; flow node `config.filter` and dashboard widget filters are author-written the same way. A report or flow authored to filter on a formula field used to run and quietly return the wrong row set; it now fails loudly, with the remedy in the message.
+
+Registered on the inherited ruling of #7095 ("register it anyway"), re-affirmed for this axis at triage on 2026-08-13 (#8370): the shape is identical to the sort axis and the consequence here is larger. #8296, #8370, #7095, #6994, #6924, #6674, ADR-0112.
+  - Done when: No filter names a `formula` field on any surface — grep your saved report definitions (`sys_saved_report.query.filter`), flow node `config.filter`, dashboard widget filters and view filters for a filtered field whose object declares it as a `formula`, and denormalise each onto a stored column written when the source changes. A `summary` / `autonumber` field needs no action: both have real maintained columns and filter correctly. Reads complete with no `INVALID_FIELD` naming a virtual `formula` field in a filter, at either door.
 - **`engine-find-formula-order-by-refused`** — `engine.find(object, { orderBy }) and engine.findOne(object, { orderBy }) naming a `formula` field — the direct engine path, not the REST ingress` → denormalise the value onto the object (a stored field, written when the source changes) and sort by that — the same remedy the REST ingress has prescribed since #6924 / #6994; a `summary` field is unaffected and still sorts, because it gets a real maintained column
   - Why not automatic: #4226 / #4256 / #6994 closed the SORT axis at the REST ingress (`assertSortFieldsExist`, `400 INVALID_SORT`), which covers everything reaching `findData`: the list route, `POST /data/:object/query`, the export route and the RPC dispatcher. A caller reaching `engine.find()` / `engine.findOne()` DIRECTLY passed through none of it, and a `formula` ORDER BY there was dropped in silence. Measured on a real driver: `asc` and `desc` came back BYTE-IDENTICAL, in insertion order, under a success, with the rows carrying the very values they were asked to be ordered by. No column exists to order by (a formula is computed on read, so no driver materialises one), so the ORDER BY reached the driver, found nothing, and the unknown-column backstop returned the rows unordered.