Feat/update diff report

CHANGELOG.md

@@ -16,6 +16,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   (`.avdl`). Reuses the exporter's existing struct/enum/dependency-resolution
   logic; removes the need to export `.avdl` and separately convert it with
   `avro-tools idl2schemata` to get `.avsc` output.
+- `--types-overlays/-L` option for all commands supporting `--types` (except `vspec export id`). Partial types files are merged into the types tree, so headers can be defined in any of the files. Modifications of existing nodes are logged as warnings.
+- `vspec diff` now emits a top-level `references` map (`{"output_type": "<struct label>"}`) on struct-typed `PROPERTY` events, declaring the dependency on the referenced struct `ENTITY`. Lets `modl sync` (ddmodl>=0.7.0) re-contract dependent properties when the referenced struct itself gets a new contract. See [diff.md](docs/diff.md#references).
+- `vspec diff` now emits a top-level `property_paths` list, declaring every concrete literal runtime path (and its instance `tags`) at which each binding-eligible property occurs, including struct-field usage-site flattening and an entity instance-delta cascade. Required by `modl sync` (ddmodl>=0.8.0). See [diff.md](docs/diff.md#property_paths).
+
+### Changed
+
+- `vspec diff` `property_paths` are now computed from the vss-tools tree (parsed and instance-expanded for both snapshots) instead of a hand-rolled expansion, so reported paths match `vspec export` output (no phantom paths for leaves declared below an instance, `instances_relation: sibling` supported). A snapshot model that cannot be expanded now fails the diff. Consistency rules between `changes` and `property_paths` are documented in [diff.md](docs/diff.md#consistency-between-changes-and-property_paths).
+- `vspec diff` no longer emits `ENTITY` events for instance branches (e.g. `Vehicle.Axle.Axle1.Wheel.Left`); instances are reported only through the `instances` aspect and `property_paths`. Events anchored below an instance (`Vehicle.Foo.Red.Extra`) are re-parented to their nearest authored ancestor, so no `ENTITY` `MODIFIED` event without a prior concept is fabricated. Fixes `modl sync` failing with `No concept found with current_label ...`.
+- Bumped `ddmodl` to `0.8.0`. `vspec diff`'s `instances`/`instances_added`/`instances_removed` aspects are now per-dimension maps (`{"dimension1": [...], "dimension2": [...]}`) instead of flat Cartesian-product path-string lists, matching modl's new dimension-map IR contract. The top-level `instantiate` field has been removed from `PROPERTY` events entirely — it has no representation in modl's IR anymore (previously-generated diff reports relying on it must be regenerated).
+
 
 ## 6.1
 

docs/diff.md

@@ -20,7 +20,8 @@ vspec diff -c snapshot_v1/
 {
   "previous": "snapshot_v1/",
   "current":  "snapshot_v2/",
-  "changes":  [ ... ]
+  "changes":  [ ... ],
+  "property_paths": [ ... ]
 }
 ```
 
@@ -37,6 +38,7 @@ Each entry in `changes` is a modl IR event. Fields present depend on `kind` and
 | `previous_aspects` | `REMOVED` only | Full prior-state snapshot of the removed element — mandatory and non-empty. |
 | `content` | `MODIFIED` `ENTITY`/`ENUMERATION_SET` | List of `{label, change_type}` summarising which children changed. |
 | `is_leaf` | every `PROPERTY` event | `true` for a primitive/scalar property (binding-eligible); `false` when the property references another entity. Mandatory on every `PROPERTY` event (`ADDED`/`REMOVED`/`MODIFIED`), reflecting the current (or, on `REMOVED`, previous) state. Forbidden on `ENUM_VALUE`. See [is_leaf](#is_leaf). |
+| `references` | struct-typed `PROPERTY` events only | `{"output_type": "<struct label>"}` declaring the struct entity this property's `output_type` points at. `ADDED`: always present for non-leaf properties. `MODIFIED`: present only when `datatype` changed — omitted otherwise to keep modl's stored edge. Forbidden on `REMOVED` and `ENUM_VALUE`. See [references](#references). |
 
 ## Kind mapping
 
@@ -89,13 +91,20 @@ event carries a wrapped `is_list` delta if it flips between `true`/`false`.
 knows which properties are eligible for binding generation: only leaf (primitive/scalar) properties
 get bindings; properties that reference another entity never do.
 
-It is derived with two rules:
+It is derived with three rules:
 
 1. **Branch/struct duality pointer events** (see [Branch duality](#branch-duality)) always get
    `is_leaf: false` — they represent a reference to another entity by construction.
-2. **Real signals and struct members** get `is_leaf` based on whether their base `datatype`
-   (stripped of any trailing `[]`) resolves to a known primitive VSS type. A `sensor`/`actuator`/
-   `attribute` whose `datatype` names a struct (e.g. `datatype: VehicleDataTypes.MyStruct`) is
+2. **Struct fields** (properties declared inside a `struct`, i.e. sourced from `structs.vspec`)
+   always get `is_leaf: false` and never `binding_eligible: true`, regardless of their own
+   `datatype` — a struct has no standalone runtime address of its own, so its fields are never
+   independently addressable and never get a `property_paths` entry either (see
+   [property_paths](#property_paths)). Only the property that references the struct (reported
+   with `binding_eligible: true`) is binding-eligible; `references`/`output_type` for a struct
+   field is still derived from its real `datatype`, unaffected by this rule.
+3. **Real signals** get `is_leaf` based on whether their base `datatype` (stripped of any
+   trailing `[]`) resolves to a known primitive VSS type. A `sensor`/`actuator`/`attribute`
+   whose `datatype` names a struct (e.g. `datatype: VehicleDataTypes.MyStruct`) is
    `is_leaf: false`, even though it isn't a duality pointer itself:
 
 ```yaml
@@ -110,6 +119,50 @@ A.RichSensor:
 
 `is_leaf` is never present on `ENUM_VALUE` events.
 
+## references
+
+`references` declares which `ENTITY` a struct-typed property's contract depends on, as a
+`{"output_type": "<struct label>"}` map. This lets `modl` cascade a new contract onto the property
+whenever the referenced struct itself gets a new contract (see modl's own
+`diff_report_template.md` for the full cascade rules), without vss-tools having to compute any
+digest or helper aspect.
+
+`output_type` is the only role vss-tools ever declares: it is the only VSS aspect whose value names
+another `ENTITY` (a struct). Units are plain opaque strings matching an `ENUM_VALUE` label, not an
+`ENTITY`/`ENUMERATION_SET` label, so they are never eligible for `references` and are not declared.
+
+- **ADDED**: always present for a non-leaf property (`is_leaf: false`), carrying the struct label
+  from `aspects.output_type`.
+- **MODIFIED**: present only when `datatype` changed in this release:
+  - the property now points at a struct (newly or retargeted to a different one) →
+    `{"output_type": "<new struct label>"}`
+  - the property now resolves to a primitive, dropping a previously-held struct reference → `{}`
+  - `datatype` unchanged → `references` is omitted entirely, keeping modl's stored edge (e.g. a
+    plain description edit on a struct-typed property never touches `references`).
+- **REMOVED**: never present — forbidden by the modl IR.
+- **`ENUM_VALUE`**: never present — forbidden by the modl IR.
+- **Branch/struct duality pointer events** (see [Branch duality](#branch-duality)): never present —
+  these events never carry `output_type` in the first place.
+
+```yaml
+A.Location:
+  type: sensor
+  datatype: Types.Position
+```
+
+```json
+{
+  "label":        "A.Location",
+  "parent_label": "A",
+  "kind":         "PROPERTY",
+  "change_type":  "ADDED",
+  "is_leaf":      false,
+  "binding_eligible": true,
+  "references":   { "output_type": "Types.Position" },
+  "aspects": { "output_type": "Types.Position", "is_list": false, "is_required": false }
+}
+```
+
 ## Aspect wrapping on MODIFIED events
 
 Every changed key in a `MODIFIED` event's `aspects` is wrapped with an `_op` annotation, per the modl
@@ -123,11 +176,17 @@ IR contract:
 }
 ```
 
-The two exceptions are the directional instance-list keys, which stay as plain unwrapped arrays since
-they carry a delta rather than a single old/new pair:
+The two exceptions are the directional instance dimension-map keys, which stay as plain unwrapped
+maps since they carry a per-dimension delta rather than a single old/new pair. `instances`,
+`instances_added`, and `instances_removed` all share the same dimension-map shape: one entry per
+instance dimension, positionally named `dimension1`, `dimension2`, ... (no Cartesian join across
+dimensions):
 
 ```json
-"aspects": { "instances_added": ["Center"], "instances_removed": ["Row3"] }
+"aspects": {
+  "instances_added":   { "dimension1": ["Center"] },
+  "instances_removed": { "dimension2": ["Row3"] }
+}
 ```
 
 ## Change types
@@ -169,6 +228,77 @@ flowchart LR
 If `fka` is missing, or the node type doesn't match, the pair is reported as independent `REMOVED` +
 `ADDED`.
 
+## property_paths
+
+Alongside `changes`, every binding-eligible `PROPERTY` (`is_leaf: true`, or `binding_eligible: true`
+for struct-typed properties) gets an entry in the top-level `property_paths` list, declaring every
+concrete literal runtime path at which it occurs:
+
+```json
+{
+  "label": "Vehicle.Axle.Wheel.Speed",
+  "parent_label": "Vehicle.Axle.Wheel",
+  "paths": [
+    { "tags": { "Vehicle.Axle": { "dimension1": "Axle1" }, "Vehicle.Axle.Wheel": { "dimension1": "Left" } },
+      "path": "Vehicle.Axle.Axle1.Wheel.Left.Speed", "change": "unchanged" },
+    { "tags": { "Vehicle.Axle": { "dimension1": "Axle1" }, "Vehicle.Axle.Wheel": { "dimension1": "Right" } },
+      "path": "Vehicle.Axle.Axle1.Wheel.Right.Speed", "change": "unchanged" }
+  ]
+}
+```
+
+- **Paths come from the vss-tools tree.** Each snapshot's `model.vspec` is parsed and
+  instance-expanded with the same tree code `vspec export` uses, for both the previous and the
+  current snapshot, and the leaf paths (`sensor`, `actuator`, `attribute`) of the two trees are
+  compared. Reported paths are therefore exactly the leaf paths an expanded export would contain:
+  instanced ancestors fan out into one entry per combination, each hop appearing after the
+  entity's own name (`Vehicle.Axle.Axle1.Wheel.Left.Speed`), `instantiate: false` pins a leaf to
+  its parent, a leaf declared below an instance (`Vehicle.Foo.Red.Extra`) exists for that instance
+  only, and `instances_relation: sibling` is followed as the tree resolves it. A snapshot model
+  the tree cannot expand fails the diff instead of producing guessed paths.
+- **`tags`** maps each instanced entity's FQN label to the `{dimension1: value, ...}`
+  combination it contributed to that path — the same dimension-map shape used for `instances`. This
+  is the canonical identity used for `change` classification: renaming an instanced ancestor changes
+  the literal `path` string but not its `tags`, so the entry stays `"unchanged"` (only `path` is
+  refreshed) rather than manufacturing a spurious add+remove pair.
+- **`change`** is `"added"`/`"removed"`/`"unchanged"` for that specific path+tags combination,
+  independent of the parent property's own `change_type` — e.g. an entity gaining a new instance
+  value re-reports every property under it with one new `"added"` path entry, even though the
+  property itself has no event of its own in `changes` (the *entity instance-delta cascade*).
+- **Struct fields never get their own entry.** A struct has no standalone runtime address of its
+  own — only the property that references the struct (reported with `binding_eligible: true`) gets
+  a `property_paths` entry, at its own path. Struct members are not flattened onto usage sites.
+
+### Consistency between `changes` and `property_paths`
+
+`changes` are computed on the authored snapshot entries, `property_paths` on the expanded trees.
+Both live in one report and are kept consistent by these rules (enforced by
+`tests/vspec/test_diff/test_diff_path_consistency.py`):
+
+1. **Completeness** — every binding-eligible model `PROPERTY` event has exactly one `property_paths`
+   entry with the same `label` and `parent_label`.
+2. **Soundness** — every entry is justified by its own event, or by an `ENTITY` `MODIFIED` event with
+   `instances_added`/`instances_removed` for an entity in the entry's `tags`.
+3. **Event to path** — `ADDED` → all paths `added`; `REMOVED` → all paths `removed`. A `MODIFIED`
+   property has `added`/`removed` paths only because a tagged entity changed its instances or the
+   property's effective `instantiate` flipped; otherwise all paths are `unchanged`.
+4. **Path to cause** — an `added` path on a property present in both snapshots requires
+   `instances_added` on a tagged entity (or an `instantiate` flip); a `removed` path requires
+   `instances_removed` (or a flip).
+5. **Path integrity** — `added`/`unchanged` paths exist in the current tree, `removed`/`unchanged`
+   paths in the previous tree (previous paths pass through the ancestor-rename map), and no path
+   is reported twice. Struct fields never have entries.
+6. **Tag validity** — tag keys are entity labels and each tag value is one of that entity's
+   instance values on the side the path belongs to.
+7. **No instance branch events** — instance branches (`Vehicle.Axle.Axle1.Wheel.Left`) are never
+   reported as entities; instances are an aspect of the authored entity and expanded paths only
+   appear in `property_paths`. Every `ENTITY`/`PROPERTY` event carries an authored label.
+8. **Parent integrity** — every `parent_label` is either an `ENTITY` event of the report or an
+   entity already authored in the previous snapshot, and every `ENTITY` `MODIFIED` event refers to
+   an entity that existed in the previous snapshot (a first run has none). A leaf declared below an
+   instance (`Vehicle.Foo.Red.Extra`) is anchored to its nearest authored ancestor
+   (`Vehicle.Foo`), never to the implicit `Vehicle.Foo.Red`.
+
 ## Example output
 
 ```json

docs/export.md

@@ -136,6 +136,18 @@ a user-defined type. If no matching type is found an error will be given.
 It is possible to use `--types/-t <file>` multiple times. Any additional files after the first one is then treated similar
 to overlay files, i.e. they are merged into previous file and it is if needed possible to redefine already defined types.
 
+Each `--types/-t` file must be self-contained. To complement the types with partial files, use
+`--types-overlays/-L <file>` (can be given multiple times). Overlays are merged into the types given with `--types`:
+
+- New types and properties are added. A new branch, struct or property needs its parent to be defined in a `--types` file or in any of the overlays, otherwise an error is given.
+- Attributes of already defined nodes are overwritten. Every modified attribute is logged as a warning.
+- All nodes must be below the root of the `--types` files. Wildcards are not supported.
+- `--types-overlays` requires `--types`. It is not available for `vspec export id`.
+
+```bash
+vspec export json --types VehicleDataTypes.vspec --types-overlays MoreTypes.vspec --vspec test.vspec --output out.json
+```
+
 Depending on exporter, type definitions may either be transformed to a separate file with structure similar to the
 "normal" output file, or integrated into the "normal" output file. If type output is given to a separate file then
 the name of that file can be specified by the `--types-output` argument.

pyproject.toml

@@ -18,7 +18,7 @@ dependencies = [
     "pandas>=2.2.3",
     "case-converter>=1.2.0",
     "inflect>=7.5.0",
-    "ddmodl==0.4.0",
+    "ddmodl>=0.8.1",
 ]
 authors = [
 {name="COVESA VSS", email="covesa-dev@covesa.global"}

src/vss_tools/cli_options.py

@@ -14,6 +14,8 @@
 from vss_tools.model import get_all_model_fields
 from vss_tools.strict import StrictOption
 
+TYPES_OVERLAYS_META_KEY = "vss_tools.types_overlays"
+
 
 def validate_attribute(value):
     """
@@ -158,6 +160,35 @@ def validate_attribute(value):
 )
 
 
+def _store_types_overlays(ctx: click.Context, param: click.Parameter, value: tuple[Path, ...]) -> tuple[Path, ...]:
+    ctx.meta[TYPES_OVERLAYS_META_KEY] = tuple(value)
+    return value
+
+
+def get_types_overlays_from_context() -> tuple[Path, ...]:
+    """
+    Returns the '--types-overlays' given on the command line.
+    Empty if not running inside a command or if the option is not available.
+    """
+    ctx = click.get_current_context(silent=True)
+    if ctx is None:
+        return ()
+    return tuple(ctx.meta.get(TYPES_OVERLAYS_META_KEY, ()))
+
+
+# The value is not passed to the command function. It is stored in the click
+# context and picked up by 'get_trees', so it can be registered on any command
+# with '--types' without touching the command itself.
+types_overlays_opt = option(
+    "--types-overlays",
+    "-L",
+    multiple=True,
+    type=click.Path(dir_okay=False, readable=True, path_type=Path, exists=True),
+    expose_value=False,
+    callback=_store_types_overlays,
+    help="Overlay files to apply on top of the types given with '--types'.",
+)
+
 types_output_opt = option(
     "--types-output",
     type=click.Path(dir_okay=False, writable=True, path_type=Path),

src/vss_tools/compose.py

@@ -67,6 +67,7 @@ def _resolve_paths(files: tuple[Path, ...], default: Path) -> list[Path]:
 @clo.quantities_opt
 @clo.units_opt
 @clo.types_opt
+@clo.types_overlays_opt
 @clo.strict_exceptions_opt
 @click.option(
     "--output-dir",
@@ -109,6 +110,7 @@ def cli(
         output_dir.mkdir(parents=True)
 
     # Phase 1: validate the full model with expansion — any real error aborts here.
+    types_overlays = clo.get_types_overlays_from_context()
     get_trees(
         vspec=vspec,
         include_dirs=include_dirs,
@@ -121,6 +123,7 @@ def cli(
         overlays=overlays,
         expand=True,
         strict_exceptions_file=strict_exceptions,
+        types_overlays=types_overlays,
     )
 
     log.info(f"Composing vspec model into {output_dir}...")
@@ -137,7 +140,8 @@ def cli(
     log.info(f"Signal tree written to {model_out}")
 
     if types:
-        raw_types = load_vspec(include_dirs_list, list(types), "Types")
+        # Types overlays are merged into the snapshot so that it is self-contained
+        raw_types = load_vspec(include_dirs_list, list(types) + list(types_overlays), "Types")
         structs_data = {fqn: _filter_raw_node(attrs, SNAPSHOT_EXCLUDE_FIELDS) for fqn, attrs in raw_types.data.items()}
         structs_out = output_dir / STRUCTS_SNAPSHOT_FILENAME
         if structs_out.exists():