feat(analytics): result envelope, chart gating, param binders, manifest fields (ISSUE-138/139/140/149) (odpi#453)

* docs(issues): log ISSUE-148 (AssetMaker action queries default to IN_PROGRESS, dropping journal entries)

Reported by the egeria-workspaces-fs session.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>

* fix(asset-maker): default activity_status_list to None (no filter) (ISSUE-148)

Seven AssetMaker action/process queries (find_processes,
get_processes_by_category, get_action_targets,
get_actions_for_action_target, get_assigned_actions,
get_actions_for_sponsor, get_actions_for_requester) defaulted to
["IN_PROGRESS"], which silently dropped journal entries and every
finished action. _async_find_processes defaulted to [] while its sync
wrapper used ["IN_PROGRESS"], so the two disagreed. All 14 signatures
now default to None.

Behaviour change: callers that want only in-progress actions must pass
["IN_PROGRESS"] explicitly. No known caller relies on the old default
(checked with the egeria-workspaces-fs and trellis sessions). Noted for
the 6.2.3 release notes in BACKLOG.md.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>

* docs(issues): log ISSUE-149/150 (analytic manifest fields, Analytic Function Dr.Egeria family); 137/141 shipped in 6.2.2

Handed off by the Resource Explorer PR/CI session on the owner's
instruction, from trellis' analytic walking-skeleton brief. A1-A5 of
the handoff were already logged as ISSUE-137..141 (137 and 141 are
fixed and released in 6.2.2).

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>

* docs(issues): record ISSUE-149 design answers from the RE handoff

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>

* docs(issues): log ISSUE-151 (get_engine_actions calls a View Server path that doesn't exist)

Reported by the RE PR/CI session; confirmed from AutomatedCurationResource
and a read-only live check. find_engine_actions("*") is the working route.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>

* fix(automated-curation): remove get_engine_actions; no View Server endpoint (ISSUE-151)

get_engine_actions posted to automated-curation/engine-actions, which
AutomatedCurationResource does not map, so it always failed with a 404
(reported as "unable to connect"). EngineAction is an Asset in 6.2, so
find_engine_actions("*") is the list-all; print_engine_actions now uses
it. No callers outside egeria-python (checked egeria-workspaces-fs and
trellis). API removal noted for the 6.2.3 release notes.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>

* feat(analytics): result envelope, chart gating, param binders, manifest fields (ISSUE-138/139/140/149)

Requested by trellis-analytics (the Analytic Library walking skeleton):
- ISSUE-149: AnalyticFunctionSpec gains optional kind (routine|step),
  source (open str; unknown values warn), version, envelope,
  annotation_type, resource_types, family and tags. Existing entries
  and trellis's manifest subclasses validate unchanged.
- ISSUE-139: the executor reads a result's state envelope (several
  carrier shapes) and returns it on every analytic output; charts get
  it as a subtitle.
- ISSUE-140: with an envelope, SERIES/BAR/PIE draw a chart only when
  the state is measured; otherwise the data and the state come back.
- ISSUE-138: ActionParameter.analytic_binders builds a non-JSON
  argument (e.g. a reader) from call params via a named factory, so a
  routine can be named by its own dotted path.
Results without an envelope behave exactly as before.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>

* fix(analytics): chart nothing_found as well as measured (ISSUE-140)

nothing_found is an established answer (a real zero), so a chart of
zeros is honest; only not_established / not_applicable, whose numbers
are unknown, are never charted. Requested by the RE PR/CI session for
trellis-analytics, whose tile already charts nothing_found.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>

* docs(issues): rewrite ISSUE-150 with the agreed direction

Declaration as a GovernanceActionType that Resource Explorer runs as an
engine host (async, recorded by Egeria); synchronous in-process runs for
screens and reports bypass Egeria and are not recorded.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>

* docs(issues): ISSUE-150 - RE engine-host answers and the Dr.Egeria gaps they expose

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>

* docs(issues): ISSUE-150 - three requirements from RE's design session

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>

---------

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>

Author: dwolfson

BACKLOG.md

@@ -16,6 +16,8 @@ Once the Egeria release with the delete fixes is published and the local quickst
 2. **Other delete issues** to re-check at the same time: ISSUE-117 (Memento cascade delete), ISSUE-125 (`deleteEnumDef`) and ISSUE-132 (`cascadedDelete` field name).
    Also ISSUE-147, the valid-metadata clears: re-run the ISSUE-147 sequence (setup then clear for a value, a map name and a map value on a made-up property). Expect no 400-011 error after a successful clear, and expect the map value to actually be removed.
 3. Update each `PYEGERIA_ISSUES.md` entry with the result, then do the held 6.2.3 release. The owner wants outstanding issues fixed before releasing.
+   **The 6.2.3 release notes must list this behaviour change:** ISSUE-148. `AssetMaker.find_processes`, `get_processes_by_category`, `get_action_targets`, `get_actions_for_action_target`, `get_assigned_actions`, `get_actions_for_sponsor` and `get_actions_for_requester` now default `activity_status_list` to `None` (no filter) instead of `["IN_PROGRESS"]`. A caller that relied on getting only in-progress actions by default now also gets completed, cancelled and status-less actions (e.g. journal entries), so it should pass `["IN_PROGRESS"]` explicitly.
+   **Also list this API removal:** ISSUE-151. `AutomatedCuration.get_engine_actions()` is removed, because the View Server has no list-all endpoint for it (it always got a 404). Use `find_engine_actions("*")` instead.
 
 ---
 

PYEGERIA_ISSUES.md

@@ -957,12 +957,40 @@ A `FormatSet` can declare either path, or both (in which case the analytic path
 
 All three chart formats are dispatched *before* the normal Format-row lookup (`exec_report_spec`), since no `FormatSet` needs to declare a `SERIES`/`BAR`/`PIE` `Format` row for this to work — only `DICT`/`JSON` (or whatever find-path formats it also supports) need a declared row.
 
+**Result state (the envelope).** A routine can say how sure it is by returning a state with its result: `measured`, `nothing_found`, `not_established` or `not_applicable`, plus a reason. The executor recognises a dict with `state`/`state_reason` (and an optional ready-made `subtitle`), a dict with `resultState`/`resultStateReason`, a nested `envelope` dict in either style, or an object whose `envelope` attribute has `result_state`/`result_state_reason`. When it finds one:
+- every output shape gets `"envelope": {"state", "reason", "subtitle"}` beside `kind`/`data`;
+- a chart gets the subtitle on it (`title: {"text", "subtitle"}`);
+- **a chart is drawn only for an established answer: `measured`, or `nothing_found`** (the routine read the record and the count really is zero, so a chart of zeros is honest). For `not_established` and `not_applicable` the numbers aren't known, so the caller gets the data and the state instead; zeros there would read as an answer.
+A result without an envelope is returned exactly as before. A result that is a pydantic model is returned as plain data (`model_dump`). See ISSUE-139/140.
+
+**Building non-JSON arguments (`analytic_binders`).** Report-spec params are JSON, and only leading parameters named `mgr`/`ce`/`expert`/`client` get an Egeria client. To name a routine that takes some other object (e.g. a reader) by its own dotted path, map the parameter to a factory:
+
+```python
+ActionParameter(
+    analytic_function="trellis_analytics.routines.curation_coverage",
+    analytic_binders={"ods": "trellis_analytics.readers.ods.OdsReader"},
+    analytic_spec_params={"resource_type": "database"},
+)
+```
+
+Here `OdsReader(url)` takes `url`, so the caller passes `url` and the routine receives the built reader as `ods`. The factory is called with the call params its own signature names, and those params are consumed unless the routine also takes them. A routine with no client parameter still connects to nothing (ISSUE-137, ISSUE-138).
+
 **`analytic_spec_params` are defaults, not pins.** Unlike the find path's `spec_params` (always fixed — a caller-supplied value for the same key is silently overridden), `analytic_spec_params` set a *default* that a caller-supplied value for the same key overrides. This is what lets a report spec like the demo "Element Count by Type" (which pins `type_name: "GlossaryTerm"` as its default) actually be re-pointed at a different type at run time instead of being permanently locked to the demo's own default.
 
 **The analytic function registry** (`pyegeria/view/analytic_registry.py`) is a separate, hand-written catalog of every analytic function available to point `analytic_function` at — `AnalyticFunctionSpec` entries carry `function` (dotted import path), `description`, `returns` (result shape), `params`, and a `generic: bool` flag with a `binding_note`:
 - `generic=True` — what the function counts/aggregates is itself a parameter (e.g. `count_elements(type_name=...)`), freely retargetable at any element type via `analytic_spec_params`.
 - `generic=False` — the vocabulary is hardcoded in the function body (e.g. `governed_coverage` always counts a fixed set of governance classifications); a report spec can still point at it, but always gets that one fixed metric.
 
+**Manifest fields (ISSUE-149).** `AnalyticFunctionSpec` also carries optional fields for analytic libraries such as trellis-analytics. All have defaults, so existing entries are unchanged:
+- `kind`: `routine` (runs in the caller's process, the default) or `step` (a survey-style step);
+- `source`: where the data comes from, as free text. Known values are `ods`, `egeria` and `resource`; any other value logs a warning and is kept;
+- `version`;
+- `envelope: bool` (the result carries a state, see above);
+- `annotation_type` (a step's result type);
+- `resource_types`;
+- `family`: the same word and meaning as `FormatSet.family`; in Egeria a family is a Collection;
+- `tags`.
+
 `get_analytic_registry()` combines a `BUILTINS` tier (the 10 `overview_metrics` functions shipped today) with optional `CONFIG`/`RUNTIME` tiers, mirroring `base_report_formats.py`'s report-spec registry shape.
 
 **Demo report specs** (`pyegeria/view/analytic_demo_specs.py`) — one real, executable `FormatSet` per registered analytic function, proving each is actually runnable as a report spec (not just documented in the registry). Named `Analytic Demo - <Heading>`, family `"Analytic Function Demo"`. Each spec's `description` states plainly whether the underlying function is generic or a fixed metric (pulled from the registry, not duplicated by hand).

docs/output-formats-and-report-specs.md

@@ -340,7 +340,7 @@ async def _async_get_xxx_by_guid(
   - **Planned**: TODO
 - [ ] `get_create_csv_data_file_element_from_template(self, file_name, file_type, file_path_name, version_identifier, file_encoding='UTF-8', file_extension='csv', file_system_name=None, description=None)`
   - **Planned**: TODO
-- [ ] `get_engine_actions(self, start_from=0, page_size=0, output_format='JSON', report_spec='EngineAction', body=None)`
+- [x] ~~`get_engine_actions(...)`~~ removed (ISSUE-151): no View Server endpoint; use `find_engine_actions("*")`
   - **Planned**: TODO
 - [x] `get_engine_actions_by_name(self, name, start_from=0, page_size=0, output_format='JSON', report_spec='EngineAction', body=None)`
   - **Planned**: Standard Get By Name Pattern

docs/parameter_cleanup_plan.md

@@ -3781,7 +3781,7 @@ def detach_software_capability_from_asset(
     async def _async_find_processes(
         self,
         search_string: str = "*",
-        activity_status_list: list[str] = [],
+        activity_status_list: Optional[list[str]] = None,
         starts_with: bool = True,
         ends_with: bool = False,
         ignore_case: bool = True,
@@ -3814,7 +3814,7 @@ async def _async_find_processes(
         ----------
         search_string: str, default = "*"
             - search string for process properties
-        activity_status_list: list[str], default = []
+        activity_status_list: list[str], default = None (no filter: every activity status, including none)
             - optional activity status list to filter by
         starts_with: bool, default = True
         ends_with: bool, default = False
@@ -3881,7 +3881,7 @@ async def _async_find_processes(
     def find_processes(
         self,
         search_string: str = "*",
-        activity_status_list: list[str] = ["IN_PROGRESS"],
+        activity_status_list: Optional[list[str]] = None,
         starts_with: bool = True,
         ends_with: bool = False,
         ignore_case: bool = True,
@@ -3914,7 +3914,7 @@ def find_processes(
         ----------
         search_string: str, default = "*"
             - the search string to use to find matching processes
-        activity_status_list: list[str], default = ["IN_PROGRESS"]
+        activity_status_list: list[str], default = None (no filter: every activity status, including none)
             - optional activity status list to filter by
         starts_with: bool, default = True
             - if True, the search string must match the start of the property value
@@ -3988,7 +3988,7 @@ def find_processes(
     async def _async_get_processes_by_category(
         self,
         category: str,
-        activity_status_list: list[str] = ["IN_PROGRESS"],
+        activity_status_list: Optional[list[str]] = None,
         graph_query_depth: int = 3,
         start_from: int = 0,
         page_size: int = 0,
@@ -4003,7 +4003,7 @@ async def _async_get_processes_by_category(
         Parameters
         ----------
         category: str
-        activity_status_list: list[str], default = ["IN_PROGRESS"]
+        activity_status_list: list[str], default = None (no filter: every activity status, including none)
         graph_query_depth: int, default = 3
             - depth for relationship traversal
         start_from: int, default = 0
@@ -4048,7 +4048,7 @@ async def _async_get_processes_by_category(
     def get_processes_by_category(
         self,
         category: str,
-        activity_status_list: list[str] = ["IN_PROGRESS"],
+        activity_status_list: Optional[list[str]] = None,
         graph_query_depth: int = 3,
         start_from: int = 0,
         page_size: int = 0,
@@ -4064,7 +4064,7 @@ def get_processes_by_category(
         ----------
         category: str
             - category name to filter by
-        activity_status_list: list[str], default = ["IN_PROGRESS"]
+        activity_status_list: list[str], default = None (no filter: every activity status, including none)
             - optional activity status list to filter by
         graph_query_depth: int, default = 3
             - depth for relationship traversal
@@ -4526,7 +4526,7 @@ def get_action_target(
     async def _async_get_action_targets(
         self,
         action_guid: str,
-        activity_status_list: list[str] = ["IN_PROGRESS"],
+        activity_status_list: Optional[list[str]] = None,
         graph_query_depth: int = 3,
         start_from: int = 0,
         page_size: int = 0,
@@ -4544,7 +4544,7 @@ async def _async_get_action_targets(
         Parameters
         ----------
         action_guid: str
-        activity_status_list: list[str], default = ["IN_PROGRESS"]
+        activity_status_list: list[str], default = None (no filter: every activity status, including none)
         graph_query_depth: int, default = 3
             - depth for relationship traversal
         body: dict | ActivityStatusRequestBody, optional
@@ -4577,7 +4577,7 @@ async def _async_get_action_targets(
     def get_action_targets(
         self,
         action_guid: str,
-        activity_status_list: list[str] = ["IN_PROGRESS"],
+        activity_status_list: Optional[list[str]] = None,
         graph_query_depth: int = 3,
         start_from: int = 0,
         page_size: int = 0,
@@ -4595,7 +4595,7 @@ def get_action_targets(
         Parameters
         ----------
         action_guid: str
-        activity_status_list: list[str], default = ["IN_PROGRESS"]
+        activity_status_list: list[str], default = None (no filter: every activity status, including none)
         graph_query_depth: int, default = 3
             - depth for relationship traversal
         start_from: int, default = 0
@@ -4641,7 +4641,7 @@ def get_action_targets(
     async def _async_get_actions_for_action_target(
         self,
         metadata_element_guid: str,
-        activity_status_list: list[str] = ["IN_PROGRESS"],
+        activity_status_list: Optional[list[str]] = None,
         graph_query_depth: int = 3,
         start_from: int = 0,
         page_size: int = 0,
@@ -4659,7 +4659,7 @@ async def _async_get_actions_for_action_target(
         Parameters
         ----------
         metadata_element_guid: str
-        activity_status_list: list[str], default = ["IN_PROGRESS"]
+        activity_status_list: list[str], default = None (no filter: every activity status, including none)
         graph_query_depth: int, default = 3
             - depth for relationship traversal
         body: dict | ActivityStatusRequestBody, optional
@@ -4692,7 +4692,7 @@ async def _async_get_actions_for_action_target(
     def get_actions_for_action_target(
         self,
         metadata_element_guid: str,
-        activity_status_list: list[str] = ["IN_PROGRESS"],
+        activity_status_list: Optional[list[str]] = None,
         graph_query_depth: int = 3,
         start_from: int = 0,
         page_size: int = 0,
@@ -4710,7 +4710,7 @@ def get_actions_for_action_target(
         Parameters
         ----------
         metadata_element_guid: str
-        activity_status_list: list[str], default = ["IN_PROGRESS"]
+        activity_status_list: list[str], default = None (no filter: every activity status, including none)
         graph_query_depth: int, default = 3
             - depth for relationship traversal
         start_from: int, default = 0
@@ -4925,7 +4925,7 @@ def unassign_action(
     async def _async_get_assigned_actions(
         self,
         actor_guid: str,
-        activity_status_list: list[str] = ["IN_PROGRESS"],
+        activity_status_list: Optional[list[str]] = None,
         graph_query_depth: int = 3,
         start_from: int = 0,
         page_size: int = 0,
@@ -4943,7 +4943,7 @@ async def _async_get_assigned_actions(
         Parameters
         ----------
         actor_guid: str
-        activity_status_list: list[str], default = ["IN_PROGRESS"]
+        activity_status_list: list[str], default = None (no filter: every activity status, including none)
         graph_query_depth: int, default = 3
             - depth for relationship traversal
         body: dict | ActivityStatusRequestBody, optional
@@ -4976,7 +4976,7 @@ async def _async_get_assigned_actions(
     def get_assigned_actions(
         self,
         actor_guid: str,
-        activity_status_list: list[str] = ["IN_PROGRESS"],
+        activity_status_list: Optional[list[str]] = None,
         graph_query_depth: int = 3,
         start_from: int = 0,
         page_size: int = 0,
@@ -4994,7 +4994,7 @@ def get_assigned_actions(
         Parameters
         ----------
         actor_guid: str
-        activity_status_list: list[str], default = ["IN_PROGRESS"]
+        activity_status_list: list[str], default = None (no filter: every activity status, including none)
         graph_query_depth: int, default = 3
             - depth for relationship traversal
         start_from: int, default = 0
@@ -5040,7 +5040,7 @@ def get_assigned_actions(
     async def _async_get_actions_for_sponsor(
         self,
         metadata_element_guid: str,
-        activity_status_list: list[str] = ["IN_PROGRESS"],
+        activity_status_list: Optional[list[str]] = None,
         graph_query_depth: int = 3,
         start_from: int = 0,
         page_size: int = 0,
@@ -5058,7 +5058,7 @@ async def _async_get_actions_for_sponsor(
         Parameters
         ----------
         metadata_element_guid: str
-        activity_status_list: list[str], default = ["IN_PROGRESS"]
+        activity_status_list: list[str], default = None (no filter: every activity status, including none)
         graph_query_depth: int, default = 3
             - depth for relationship traversal
         body: dict | ActivityStatusRequestBody, optional
@@ -5091,7 +5091,7 @@ async def _async_get_actions_for_sponsor(
     def get_actions_for_sponsor(
         self,
         metadata_element_guid: str,
-        activity_status_list: list[str] = ["IN_PROGRESS"],
+        activity_status_list: Optional[list[str]] = None,
         graph_query_depth: int = 3,
         start_from: int = 0,
         page_size: int = 0,
@@ -5109,7 +5109,7 @@ def get_actions_for_sponsor(
         Parameters
         ----------
         metadata_element_guid: str
-        activity_status_list: list[str], default = ["IN_PROGRESS"]
+        activity_status_list: list[str], default = None (no filter: every activity status, including none)
         graph_query_depth: int, default = 3
             - depth for relationship traversal
         start_from: int, default = 0
@@ -5155,7 +5155,7 @@ def get_actions_for_sponsor(
     async def _async_get_actions_for_requester(
         self,
         metadata_element_guid: str,
-        activity_status_list: list[str] = ["IN_PROGRESS"],
+        activity_status_list: Optional[list[str]] = None,
         graph_query_depth: int = 3,
         start_from: int = 0,
         page_size: int = 0,
@@ -5173,7 +5173,7 @@ async def _async_get_actions_for_requester(
         Parameters
         ----------
         metadata_element_guid: str
-        activity_status_list: list[str], default = ["IN_PROGRESS"]
+        activity_status_list: list[str], default = None (no filter: every activity status, including none)
         graph_query_depth: int, default = 3
             - depth for relationship traversal
         body: dict | ActivityStatusRequestBody, optional
@@ -5206,7 +5206,7 @@ async def _async_get_actions_for_requester(
     def get_actions_for_requester(
         self,
         metadata_element_guid: str,
-        activity_status_list: list[str] = ["IN_PROGRESS"],
+        activity_status_list: Optional[list[str]] = None,
         graph_query_depth: int = 3,
         start_from: int = 0,
         page_size: int = 0,
@@ -5224,7 +5224,7 @@ def get_actions_for_requester(
         Parameters
         ----------
         metadata_element_guid: str
-        activity_status_list: list[str], default = ["IN_PROGRESS"]
+        activity_status_list: list[str], default = None (no filter: every activity status, including none)
         graph_query_depth: int, default = 3
             - depth for relationship traversal
         start_from: int, default = 0