Merge origin/main into claude/issue-20887-analytics-nested-relation

main's 975b248 touched packages/services/service-analytics (the dataset
compiler and one of its pins), this card's surface, so it is merged before
patch round 2.

Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H
Co-authored-by: Claude <noreply@anthropic.com>

Author: claude

.changeset/18386-template-required-option-default.md

@@ -0,0 +1,12 @@
+---
+'@objectstack/rest': patch
+---
+
+fix(rest): the import template (`GET /api/v1/data/:object/export?template=true`) puts ` *` on a column only when the import refuses a row that leaves it blank
+
+Clause-②: no
+
+- A required field whose option list marks an option `default: true` no longer gets ` *` in the header, and the instructions sheet lists it as not required. A blank cell in that column is imported as the marked option.
+- A required field that declares `defaultValue: null` now gets ` *`, unless one of its options is marked `default: true`: the import treats `null` as no default and refuses the blank. A required field whose default is `''`, or `[]` on a multi-valued field, now gets ` *` too: the required check refuses that default.
+- A required `system`, `readonly` or `autonumber` field named in `?fields=` no longer gets ` *`: the import does not refuse a blank in it.
+- Each dropdown's error title, which is the column header, is cut to 32 characters, the longest title Excel's data-validation dialog takes.

.changeset/20281-connector-sync-moved-to-mapping.md

@@ -0,0 +1,83 @@
+---
+'@objectstack/spec': minor
+---
+
+feat(spec)!: connector-attached sync leaves the connector — `syncConfig` and `fieldMappings` are retired, and a `mapping` gains the `connectorSource` pull binding (#20281)
+
+**BREAKING** — `connector.syncConfig` (the `DataSyncConfig` block: `strategy`,
+`direction`, `realtimeSync`, `timestampField`, `conflictResolution`, `batchSize`,
+`deleteMode`, `filters`) and `connector.fieldMappings` (the `ConnectorFieldMapping`
+list: `source`, `target`, `defaultValue`, `dataType`, `required`, `syncMode`) are
+removed from `ConnectorSchema` and `DeclarativeConnectorEntrySchema` — so from
+`defineConnector`, `stack.connectors[]`, the `PUT /api/v1/meta/connector/:name` door
+and `AutomationEngine.registerConnector`. The `DataSyncConfigSchema`,
+`SyncStrategySchema`, `ConnectorConflictResolutionSchema` and
+`ConnectorFieldMappingSchema` exports (and their `DataSyncConfig`, `SyncStrategy`,
+`ConnectorConflictResolution`, `ConnectorFieldMapping` types and `…Parsed` aliases)
+leave `@objectstack/spec/integration` with them. ADR-0049, ruled ENFORCE on the
+maintainer's criterion for a declared-but-unenforced family, with the definition
+MOVED: every mainstream platform binds a sync to its TARGET, not to the connection.
+
+Measured before removal: no engine ever ran a connector-attached sync or moved a
+value through a connector field mapping — outside the spec package `syncConfig`
+appeared only in two comments and `fieldMappings` nowhere, the automation service's
+declared-connector item carried neither key, and the def a provider registers is its
+own. The `latest_wins` and `soft_delete` defaults read as configured policy and did
+nothing; the platform has no soft delete.
+
+**Added (declared, not yet executed):** `mapping.connectorSource` — the pull
+binding on the target side, beside the mapping's existing `targetObject`,
+`fieldMapping`, `mode` and `upsertKey`: `connector` (a `rest` or `openapi`
+connector instance), `action` (the action that reads the records), optional
+`input`, optional `recordsPath`, and an optional `watermark` (`field` on the
+record, `param` on the request) for a timestamp-incremental pull. Version 1 is a
+one-way pull. It carries no cadence (a `job` sets that), no credential (the
+connector instance holds it) and no delete or conflict policy. Nothing executes it
+in this release, and `os validate` / `os build` warn when it is authored.
+
+### FROM → TO
+
+| removed | what to write instead |
+| --- | --- |
+| `connector.syncConfig` | delete the key. A sync you still want is a `mapping` on its target object, with `connectorSource` naming the connector and its read action, `watermark` for an incremental pull, and a `job` for the cadence. `direction`, `conflictResolution` and `deleteMode` have no counterpart: the pull is one-way and writes through `mode` / `upsertKey`. |
+| `connector.fieldMappings` | delete the key, and carry its `source` → `target` pairs into that mapping's `fieldMapping` (a `defaultValue` becomes a `constant` transform). |
+| `DataSyncConfigSchema`, `SyncStrategySchema`, `ConnectorConflictResolutionSchema`, `ConnectorFieldMappingSchema` and their types | no replacement — nothing parsed a sync or a connector field mapping into anything that ran. |
+
+**The one-line fix: delete `syncConfig:` and `fieldMappings:` from every connector.**
+`os migrate meta --from 17` lists the mechanical edits for existing sources.
+
+⚠️ Runtime behaviour is deliberately **unchanged**: no connector sync ever ran.
+What changes is the answer an author gets — both keys are refused at parse with a
+prescription naming the target-side binding, and in `tsc` (their input type is
+`never`), instead of being saved with no effect.
+
+### The retirement kit
+
+- **Tombstones.** `syncConfig` and `fieldMappings` are `retiredKey()` tombstones on
+  the private `ConnectorBaseSchema` both published carriers wrap (the schema is not
+  `.strict()`, so a bare deletion would be a silent strip, ADR-0104).
+  `RETIRED_KEYS_BY_MAJOR[18]`: `integration/Connector:syncConfig`,
+  `integration/Connector:fieldMappings`,
+  `integration/DeclarativeConnectorEntry:syncConfig` and
+  `integration/DeclarativeConnectorEntry:fieldMappings`. Neither key had a default,
+  so no retired-default residue is owed.
+- **Four defs leave whole** (`RETIRED_DEFS_BY_MAJOR[18]`): `integration/DataSyncConfig`,
+  `integration/SyncStrategy`, `integration/ConnectorConflictResolution`,
+  `integration/ConnectorFieldMapping`. The two `RENAMED_DEFS` entries that targeted
+  them left the rename table.
+- **D2 conversion `connector-sync-keys-removed`** (step 18, retired from the load
+  path): strips both keys from `connectors[]` and from stored `sys_metadata`
+  connector rows (the rehydration seam replays it), one notice per key, as a
+  lossless delete. It never writes a `mapping`: a pulled mapping would start writes
+  that never happened.
+- **D3 entry `connector-sync-keys-retired`** carries the family's judgement: which
+  syncs should now exist as target-side mappings, and what an author who relied on
+  `export`, `bidirectional`, `soft_delete` or a conflict policy does without them.
+- **No deprecation window**, per the project's startup-stage posture.
+
+⚠️ **The out-of-repo consumer population is NOT MEASURED.** `@objectstack/spec` is
+published, so this is breaking for consumers no telemetry was consulted for.
+
+Clause-②: yes (narrowing)
+
+<!-- adr-0087: registered connector-sync-keys-removed, connector-sync-keys-retired -->

.changeset/20701-import-row-field-and-missing-column-wording.md

@@ -0,0 +1,23 @@
+---
+'@objectstack/rest': patch
+---
+
+fix(rest): an import row names the column the engine refused, and a missing database column is no longer called an unknown field (#20701)
+
+**`POST /api/v1/data/:object/import` — a failed row names its column.** A row the
+engine refuses with `INVALID_FIELD` (a column that names no field of the object)
+now carries `field` with that column, on the dry run and on the commit alike. It
+used to carry only `code: 'INVALID_FIELD'`, while `POST /api/v1/data/:object`
+named the field for the same key. The row reads the error's own `field` when no
+field-level finding names one; a field-level finding still wins. A unique
+conflict row now names its column too, when the database said which column it
+was, as the `409` does.
+
+**A missing database column says what the database said.** When the database
+reports that a table has no column for a field, the `400 INVALID_FIELD` answer
+now reads "The database table of object 'X' has no column for field 'f'. If the
+object declares 'f', its database schema has drifted from the metadata: run
+'os migrate' to reconcile." It used to read "Unknown field 'f'", which was false
+for a field the object declares: the engine refuses an undeclared key itself,
+before the database is reached, and keeps its own "Unknown field" wording for
+that case. `code`, `status`, `field` and `object` are unchanged.

.changeset/20808-json-stored-group-distinct-refused.md

@@ -0,0 +1,33 @@
+---
+"@objectstack/objectql": minor
+"@objectstack/spec": minor
+"@objectstack/lint": patch
+"@objectstack/service-analytics": patch
+---
+
+fix(objectql,spec)!: a `groupBy` on a multi-value field and a `count_distinct` on a JSON-stored field are refused with `INVALID_FIELD` / 400 at the engine's `aggregate`, on every driver, and the aggregate × field-type table stops accepting `count_distinct` over the JSON-stored types
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (already-registered dataset-measure-aggregate-field-type-refused) the one metadata-facing half of this change is a row of AGGREGATE_FIELD_TYPE_COMPATIBILITY narrowing, and that family's hand-migration is already registered under protocol major 18 by this id: "an aggregate the field's type accepts, per AGGREGATE_FIELD_TYPE_COMPATIBILITY", with every refused pair of the table refused at the compile door. The non-temporal sum / avg narrowing rode the same id the same way; this diff amends that entry's surface and acceptance prose to name the count_distinct rider, and corrects the min / max entry's route that called count_distinct valid over every type. The engine-door halves refuse a query shape, not a stored one: no authorable key, export or stored row moves. -->
+
+**BREAKING** (`@objectstack/objectql`): this narrows what `aggregate` accepts, in two positions, on every driver and for every caller that reaches the engine (the REST query door, a flow or hook, and the analytics strategy that lowers a cube query onto `engine.aggregate`). Shipped as `minor` under the launch-window convention for accept-set narrowings. No export or published type changes.
+
+- A `groupBy` entry that names a **multi-value** field: an inherently-multi option type (`multiselect`, `checkboxes`, `tags`), or a `select`, `lookup`, `user`, `file` or `image` field declared `multiple: true`. Both entry spellings are judged, the field name and the `{ field }` object.
+- A `count_distinct` aggregation over a **JSON-stored** field: a structured-JSON type (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`), an inherently-multi option type, or a multi-capable field declared `multiple: true`.
+
+**BREAKING** (`@objectstack/spec`): `AGGREGATE_FIELD_TYPE_COMPATIBILITY.count_distinct` no longer lists the ten JSON-stored types (the structured-JSON seven and `multiselect`, `checkboxes`, `tags`), so `isAggregateCompatibleWithFieldType('count_distinct', type)` answers `false` for them. Every reader of the table refuses those pairs now: the dataset-measure lint rule (`measure-aggregate-field-type-refused`, run by `os validate` and at a runtime dataset save), the analytics dataset compile leg (`400 DATASET_INVALID`), and the engine door above. The `count` row is unchanged.
+
+**What an author sees now.** `400 INVALID_FIELD`, naming the position (`groupBy[0]`, `groupBy[0].field`, or `aggregations[0].field`), the field and its declaration, saying the query was not run, and naming the route inside the first 500 characters the REST door keeps. For a multi-value field the route is to filter by one member: `where` with `$contains` on the field, one query per member. For a structured-JSON field it is to store the part you count in a field of its own, or to count rows with `count`. The thrown error carries `field`, `fields`, `object` and `param` (`groupBy` or `aggregations`).
+
+**Why a refusal.** Every SQL driver stores these values in a JSON column, and the drivers share no meaning for one as a group key or a distinct key. Measured through `POST /api/v1/data/:object/query` over three rows: grouping by any of the eight multi-value declarations answered one group per array on the in-memory driver, one group per serialized array on SQLite, and 500 `DATABASE_ERROR` on PostgreSQL 16. `count_distinct` over any structured-JSON or multi-value field answered 3 on the in-memory driver (equal values counted apart), 2 on SQLite (serialized text compared), and 500 on PostgreSQL (no equality operator for `json`). No example app and no published stack groups by a multi-value field or counts one distinct, so no meaning is defined for either here.
+
+**What to write instead.** A dataset measure or a query that counted a JSON-stored field distinct: use `count` over it, or store the scalar part you meant to count in a field of its own and `count_distinct` that field. A grouping by a multi-value field: filter by each member with `$contains` and count.
+
+**Who is affected.** A caller that grouped by a multi-value field, or counted a JSON-stored field distinct, on the in-memory driver or on SQLite and read the answer as a real one; on PostgreSQL both were already a 500. A dataset whose measure pairs `count_distinct` with a JSON-stored field is refused by the lint rule and the compile leg.
+
+**Unchanged.** (Two shapes the structured-JSON `groupBy` entry of this same release lists as unchanged are narrowed here: a `multiple: true` select as a group key, and `count_distinct` over a structured-JSON field. This entry is the later word on both.) A `groupBy` or `count_distinct` on a scalar-stored field, a single-value `select` or `lookup` included; `count` over any field; the `having`, filter and sort positions; and an undeclared name, which the REST door answers `INVALID_FIELD` as unknown before the engine is reached.
+
+`@objectstack/lint`: the dataset-measure refusal's hint no longer says `count_distinct` accepts every type.
+
+`@objectstack/service-analytics`: the dataset compile leg's refusal of a `count_distinct` measure over a JSON-stored field says why it diverges (the drivers compare the values for equality three ways) and prescribes `count`, or a scalar field for the part being counted; its other refusals no longer say `count_distinct` accepts every type.

.changeset/20862-automation-doors-persist.md

@@ -0,0 +1,25 @@
+---
+'@objectstack/runtime': minor
+---
+
+fix(runtime)!: the /automation create and update doors save the flow as a tenant row, so what they answer 200 for survives a restart, and the removal door deletes that row too (#20862)
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (no-migration-prescription) No metadata changes shape and nothing an author wrote is renamed or removed, so `objectstack migrate meta` has nothing to rewrite. What moves is which definition writes and removals the /automation doors accept: the ones the metadata store refuses are now refused there too. -->
+
+**BREAKING**: shipped as `minor` under the launch-window convention. `POST /api/v1/automation` and `PUT /api/v1/automation/:name` now refuse a definition the metadata store refuses, which they used to register and answer `200` for. `DELETE /api/v1/automation/:name` now relays a store's refusal to delete the flow's row.
+
+**What changed.** The create and update doors registered a flow in the automation engine and wrote no metadata row. The next boot binds flows from the stored metadata, so a flow created through `POST /automation` was gone after a restart, and an update through `PUT /automation/:name` to a flow stored through `/meta` lost to the stored definition. Both doors now save the definition through the metadata protocol's own `saveMetaItem`: the save `PUT /api/v1/meta/flow/:name` uses, and the one the clone door (`POST /automation/:name/clone`) already used. The row is env-wide and live (`active`), so the flow reads back on `/meta` and survives a cold boot. The three doors share one path.
+
+- **Engine first, store second.** The engine's registration is still the first check. Its refusal is answered as before (`400 VALIDATION_FAILED`), and nothing is saved.
+- **A save the store refuses is relayed with its own code and status, and leaves no registration behind.** A create is withdrawn from the engine. An update puts back the definition the engine held, so a refused update does not take the flow down.
+- **`DELETE /automation/:name` deletes the tenant row too**, through `deleteMetaItem`, so a flow created through the door does not come back at the next boot. The engine's own removal refusal (`DELETE_RESTRICTED` / `409`) is still raised before the store is touched. A name with no stored row is removed as before. A delete the store refuses puts the definition back and relays the refusal.
+- **Unchanged:** the locked-base refusal on a packaged flow's name (`403 NOT_OVERRIDABLE`) and the refusal of a definition claiming a package's provenance (`422 INVALID_METADATA`) still answer first. A composition with no metadata protocol keeps the engine-only registration and removal it always had.
+
+**Newly refused, because the metadata store refuses them** (measured on the showcase):
+
+- A flow name with a leading underscore. `FlowSchema` admits it and the metadata item-name grammar does not, so the door answers `400 INVALID_REQUEST`. Rename the flow to a name that starts with a letter.
+- A definition a gating runtime publish rule refuses, such as a default edge that also carries a condition (`flow-default-edge-with-condition`). The door answers `422 INVALID_METADATA` with the rule's finding. Fix the definition as the finding says.
+
+Such a flow could never be stored, so before this change it ran only until the next restart.

content/docs/deployment/validating-metadata.mdx

@@ -206,9 +206,11 @@ measures: [{ name: 'avg_closed', aggregate: 'avg', field: 'closed_at' }]
 `avg` over a temporal column is where this bites hardest: one SQL family coerces
 the stored text and returns a plausible number (the average *year*), another has
 no such function and fails at query time. `min`/`max` over the same field are
-**accepted** — they return a real instant of the field's own type — and
-`count`/`count_distinct` are accepted over every type, because they read no
-arithmetic off the value. The analytics service refuses the same pair with
+**accepted** — they return a real instant of the field's own type — `count` is
+accepted over every type, because it reads no value, and `count_distinct` over
+every type but the JSON-stored ones (the structured-JSON types and
+`multiselect` / `checkboxes` / `tags`), whose values no two backends compare
+for equality alike. The analytics service refuses the same pair with
 `400 DATASET_INVALID` when a query is built; this is the identical verdict,
 from the identical table, one door earlier. The rule stays silent wherever the
 field's type cannot be resolved (an object this stack does not define, a