feat(eval): add eval measurements and EM-001 trace_fitness scorer

action.yml

@@ -409,6 +409,28 @@ runs:
           "${STATUS_FLAGS[@]+"${STATUS_FLAGS[@]}"}" \
           "${MINT_FLAGS[@]+"${MINT_FLAGS[@]}"}"
 
+    # Eval measurements (fail-open): score run-telemetry.jsonl with the agents
+    # measurement manifest. Same job as fullsend run; never fails the agent.
+    # Scores always land in eval-measurements.jsonl (tool-agnostic artifact).
+    - name: Eval measurements
+      if: always() && inputs.agent != '__install_only__'
+      continue-on-error: true
+      shell: bash
+      env:
+        AGENT: ${{ inputs.agent }}
+        FULLSEND_DIR: ${{ inputs.fullsend-dir }}
+        # For GetRef of agents@v0 (SHA pin). Not sent to raw.githubusercontent.com.
+        GH_TOKEN: ${{ inputs.github_token }}
+      run: |
+        set -euo pipefail
+        FULLSEND_DIR="${FULLSEND_DIR:-${GITHUB_WORKSPACE}}"
+        # Binary resolves the SHA-pinned agents manifest (allowlist/hash/audit)
+        # and scores only platform run-telemetry.jsonl at the top of each runDir.
+        fullsend eval-measure \
+          --agent "${AGENT}" \
+          --fullsend-dir "${FULLSEND_DIR}" \
+          --output-dir "${GITHUB_WORKSPACE}/output"
+
     - name: Upload fullsend artifacts
       if: always() && inputs.agent != '__install_only__'
       uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1

docs/.vitepress/config.ts

@@ -260,6 +260,7 @@ export default defineConfig({
             { text: "Standalone Mint", link: "/guides/infrastructure/standalone-mint" },
             { text: "Private Repositories", link: "/guides/infrastructure/private-repositories" },
             { text: "Tracing Reference", link: "/guides/infrastructure/distributed-tracing" },
+            { text: "Eval Measurements", link: "/guides/infrastructure/eval-measurements" },
             { text: "Advanced Setup", link: "/guides/infrastructure/advanced-setup" },
             {
               text: "Layered Config Reference",

docs/ADRs/0050-distributed-tracing-instrumentation.md

@@ -155,3 +155,11 @@ and agent spans in `run-telemetry.jsonl`, which became the sole Level 1
 artifact. OTLP export also changed from post-hoc directory upload to live
 span export via the OTel SDK's batch processor. The core decision (three-level
 opt-in, OTel-native, W3C propagation) is unchanged.
+
+**2026-08-10 — Eval measurements ([ADR 0087](0087-eval-measurements-online-trace-scoring.md)):**
+online scoring of wild-run traces always writes `eval-measurements.jsonl`
+beside telemetry (tool-agnostic). Distinct from functional eval fixtures
+([ADR 0051](0051-agent-eval-harness-for-test-infrastructure.md)).
+
+> **Planned:** portable remote score export follows the same OTLP
+> configuration as this ADR — no vendor score adapters in core.

docs/ADRs/0087-eval-measurements-online-trace-scoring.md

@@ -0,0 +1,128 @@
+---
+title: "87. Eval measurements as online trace scoring with portable export"
+status: Accepted
+relates_to:
+  - operational-observability
+  - testing-agents
+topics:
+  - observability
+  - evaluation
+  - opentelemetry
+---
+
+# 87. Eval measurements as online trace scoring with portable export
+
+Date: 2026-08-10
+
+## Status
+
+Accepted
+
+## Context
+
+Agent runs already emit OpenTelemetry traces as `run-telemetry.jsonl`, with
+optional live OTLP export when `OTEL_EXPORTER_OTLP_*` is set
+([ADR 0050](0050-distributed-tracing-instrumentation.md)). Separately,
+[ADR 0051](0051-agent-eval-harness-for-test-infrastructure.md) owns the
+**functional** eval harness: curated fixtures / scenarios in
+`fullsend-ai/agents` `eval/<agent>/` that gate agent PRs. Those fixtures do
+not score wild production runs.
+
+Operators also need an **online / trend** layer on wild traces (completeness
+first; quality signals later). Fullsend must stay **backend-agnostic**: orgs
+already choose Phoenix, MLflow, Jaeger, or another OTLP collector for traces.
+Baking a single product’s Assessments/Quality API into the core CLI or managed
+workflows would force a tool decision on every install.
+
+Adjacent telemetry proposals (not competing with this score path):
+
+- **Level 3 content capture** ([#5947](https://github.com/fullsend-ai/fullsend/pull/5947)
+  — proposed ADRs 0084/0085): richer span content enables later scorers;
+  first ship reads Level 1/2 metadata in `run-telemetry.jsonl`. Measure CLI is
+  host-side after sandbox exit.
+- **Span status from run outcome** ([#5944](https://github.com/fullsend-ai/fullsend/pull/5944)):
+  OTLP Status (and `fullsend.transcript_error`) become the reliable
+  success/failure signal. EM-001 only checks that `exit_code` is **present**
+  (fitness). Outcome scorers must key on Status, not `exit_code == 0`.
+- **Observer / lessons → fixtures** ([#2423](https://github.com/fullsend-ai/fullsend/pull/2423)):
+  narrative analysis and golden-set promotion. This ADR is same-job
+  deterministic scoring on traces.
+- **Harness snapshot / forge join keys** ([#5524](https://github.com/fullsend-ai/fullsend/pull/5524)):
+  sibling artifact for harness fingerprint and
+  forge/CI pointers beside telemetry. Complementary join/identity layer;
+  primary run facts belong on the OTEL trace (Level 1), while measurements
+  stay a derived sibling file.
+
+## Options
+
+1. **Local JSONL only** — portable offline artifact; no remote scores from
+   fullsend itself.
+2. **Backend-native APIs in core** (e.g. one vendor’s Assessments API) —
+   couples every managed workflow to that product’s auth and schema.
+3. **Local JSONL + same OTLP path as agent traces for remote** — scores travel
+   with the endpoint/headers orgs already configure for ADR 0050; no second
+   vendor stack in core.
+
+## Decision
+
+Introduce **eval measurements**: deterministic scorers that read
+`run-telemetry.jsonl` after `fullsend run` in the **same** managed job
+(`fullsend eval-measure` in `action.yml`), **fail-open**. Functional eval
+scenarios remain ADR 0051 / `eval/<agent>/`; measurements never block
+delivery.
+
+In plain terms: eval measurements are the concept of scoring traces.
+[OTEL primary facts](../glossary.md#otel-primary-facts) are what happened
+on the run (the OTEL trace / `run-telemetry.jsonl`).
+[OTEL derived products](../glossary.md#otel-derived-products) are scores
+computed from that trace (`eval-measurements.jsonl`). Measurements never
+rewrite primary facts, and they are [fail-open](../glossary.md#fail-open).
+
+Scores always land in a tool-agnostic `eval-measurements.jsonl` (plus a
+small idempotency ledger) next to `run-telemetry.jsonl`. Remote score export
+will use the same `OTEL_EXPORTER_OTLP_*` configuration as ADR 0050 — no
+vendor-specific score adapters in core. `fullsend` owns the parser, scorers,
+CLI, and GHA step; `fullsend-ai/agents` owns per-agent measurement manifests
+(`eval/measurements/<agent>.yaml`) that declare which scorers to enable.
+Stock-agent defaults resolve from `agents@v0` at runtime; local files are for
+override, opt-out, or custom agents only.
+
+The first scorer is `trace_fitness` (catalog id `em-001`) — span-tree and
+attribute fitness so later scorers can trust the trace. EM-001 reads
+experimental OpenTelemetry GenAI attribute names (`gen_ai.*` constants in
+`internal/evalmeasure`); an upstream rename is an `em-001` version bump.
+
+### Versioning (per measurement, not platform “v1”)
+
+There is no product-wide “eval measurements v1” switch. “First ship” just
+means only one scorer is enabled yet. Each manifest entry carries:
+
+| Field | Meaning |
+|---|---|
+| `id` | Stable catalog id (`em-001`). New measurement concept → new id. |
+| `scorer` | Go dispatch name (`trace_fitness`). |
+| `version` | Integer **contract** version of that measurement’s checks / pass rule. |
+
+Scores and the idempotency ledger key on `id@version` (e.g. `em-001@1`).
+Bump `version` when pass/fail semantics change so trends do not mix eras.
+Add a check that does not change the pass definition → same version is fine.
+Entirely new signal → new `em-NNN` (and usually a new `scorer` string).
+
+## Consequences
+
+- Every measured run produces a reviewable, backend-agnostic score file beside
+  telemetry; missing manifests skip cleanly and measure failure never fails
+  the agent job. GitHub Actions is the first-ship managed path (uploads
+  `output/`). GitLab CI calls the same fail-open `eval-measure` CLI, writing
+  under `/tmp/fullsend-output` with no `artifacts:` block by default.
+- Core stays tool-agnostic: no product-specific score env vars in managed
+  workflows; remote scores follow OTEL when that path lands.
+- Functional scenarios (gate) and eval measurements (trend) stay separate;
+  retro can recommend either a manifest scorer or a scenario fixture.
+- Richer telemetry (Level 3 / Status fixes) expands what scorers *can* assert;
+  it does not replace this same-job path.
+- Per-measurement versioning (`id@version`) lets pass/fail semantics evolve
+  without mixing trend eras.
+- Pre-script skipped runs (`fullsend.prescript.skipped=true` on the root span)
+  are excluded from EM-001: the scorer writes `label: skip` instead of failing
+  a run that never created a sandbox.

docs/architecture.md

@@ -287,11 +287,15 @@ Observability is a cross-cutting concern that touches every other component. Eac
 - JSONL reasoning trace exposure: raw JSONL conversation transcripts are extracted from sandboxes and stored with owner-scoped access. Credential scanning acts as an invariant check on [ADR 0017](ADRs/0017-credential-isolation-for-sandboxed-agents.md)'s isolation model. Agents handling data from protected sources beyond the target repo can opt in to JSONL suppression via configuration ([ADR 0021](ADRs/0021-jsonl-reasoning-trace-exposure.md)).
 - Event-driven stage dispatch remains traceable end-to-end in the GitHub Actions UI by using synchronous `workflow_call` dispatch (see [ADR 0041](ADRs/0041-synchronous-workflow-call-event-dispatch.md)).
 - Distributed tracing: framework-native OpenTelemetry instrumentation with zero-configuration baseline. Every run produces `run-telemetry.jsonl` locally; optional live OTLP export to any compatible backend. W3C trace context propagation links multi-agent pipelines into unified traces. OTEL GenAI semantic conventions enable LLM-aware backends ([ADR 0050](ADRs/0050-distributed-tracing-instrumentation.md)).
+- Eval measurements: the concept of scoring traces ([fail-open](glossary.md#fail-open)). [OTEL primary facts](glossary.md#otel-primary-facts) stay on the run trace (`run-telemetry.jsonl`); [OTEL derived products](glossary.md#otel-derived-products) are the scores (`eval-measurements.jsonl`) ([ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md)). See [Eval Measurements](guides/infrastructure/eval-measurements.md).
+
+  > **Planned:** portable remote score export via the same OTLP configuration as agent traces ([ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md)). Not yet implemented.
 
 **Open questions:**
 
 - What signals matter most — cost, latency, token usage, action logs, decision traces, or something else?
 - ~~How do we balance detailed tracing (useful for debugging) with the volume of data agents will produce?~~ Decided in [ADR 0050](ADRs/0050-distributed-tracing-instrumentation.md): instrument all lifecycle steps comprehensively; volume is managed by backends not by suppressing data at the source.
+- ~~How do we score wild agent traces for trends without a second export stack?~~ Decided in [ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md): eval measurements write local JSONL beside telemetry; portable remote export uses the same OTLP config as traces (planned); local eval-measurements.jsonl is always written.
 - What is the retention and access model for agent logs? Who can see what? (JSONL trace access model decided in [ADR 0021](ADRs/0021-jsonl-reasoning-trace-exposure.md); retention policy and broader log access remain open.)
 - How does observability interact with the security requirement that "every action is logged, attributable, and reviewable"? (See [security-threat-model.md](problems/security-threat-model.md).)
 - Is there a real-time monitoring requirement (agent is stuck, agent is behaving anomalously), or is observability primarily forensic?

docs/glossary.md

@@ -88,20 +88,25 @@ See [autonomy-spectrum.md](problems/autonomy-spectrum.md) and [agent-architectur
 
 ### Eval Measurement
 
-A score, judge, or metric applied to agent (or agent-chain) behavior — for example cost per run, whether the code agent later passes review, or whether a review agent recommends merge and a human still intervenes. Measurements are not the inputs under test; they are what you score. The same measurement can be applied to curated [eval scenarios](#eval-scenario) or to live ("wild") production traffic, at agent scope or across the platform chain. Prefer this term (or synonyms *eval score* / *eval judge*) over the bare word "evals," which is ambiguous with [eval scenarios](#eval-scenario).
-See [testing-agents.md](problems/testing-agents.md) and [Observability](#observability).
+The concept of **scoring traces** — a score, judge, or metric applied to agent (or agent-chain) behavior. Example signals: cost per run, whether the code agent later passes review, or whether a review agent recommends merge and a human still intervenes. Measurements are not the inputs under test; they are an [OTEL derived product](#otel-derived-products) computed from [OTEL primary facts](#otel-primary-facts). Online / trend scoring of wild-run traces is decided in [ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md) (`fullsend eval-measure`, `eval-measurements.jsonl`) and is [fail-open](#fail-open). The same measurement *idea* can also be applied to curated [eval scenarios](#eval-scenario), but those PR-gate fixtures are a separate path ([ADR 0051](ADRs/0051-agent-eval-harness-for-test-infrastructure.md)). Prefer this term (or synonyms *eval score* / *eval judge*) over the bare word "evals," which is ambiguous with [eval scenarios](#eval-scenario).
+See [Eval Measurements](guides/infrastructure/eval-measurements.md), [testing-agents.md](problems/testing-agents.md), and [Observability](#observability).
 
 ### Eval Scenario
 
-A fixed, reproducible test case — a concrete input with an expected outcome that you re-run when an agent changes. Example: triage is presented with an issue asking to add a cheeseburger to the README and is expected to reject and close it. Scenarios are maintained like tests: if intentional agent behavior changes, update the scenario expectations. They answer "did this change make the agent better or worse on known cases?" and can later grow by promoting interesting production cases from telemetry into the curated set. Distinct from [eval measurements](#eval-measurement) (the scores/judges applied to a scenario or to wild traffic). Prefer this term over the bare word "evals."
-See [testing-agents.md](problems/testing-agents.md) (golden-set evaluation).
+A fixed, reproducible test case — a concrete input with an expected outcome that you re-run when an agent changes. Example: triage is presented with an issue asking to add a cheeseburger to the README and is expected to reject and close it. Scenarios are maintained like tests: if intentional agent behavior changes, update the scenario expectations. They answer "did this change make the agent better or worse on known cases?" and can later grow by promoting interesting production cases from telemetry into the curated set. Distinct from [eval measurements](#eval-measurement) (online/trend scores on wild traces, or judges applied to a scenario). Prefer this term over the bare word "evals." Also called a *functional eval fixture* in agent CI.
+See [ADR 0051](ADRs/0051-agent-eval-harness-for-test-infrastructure.md) and [testing-agents.md](problems/testing-agents.md) (golden-set evaluation).
 
 ### Evergreen
 
 A workflow concept where a repository automatically stays up-to-date with dependency updates (e.g., Renovate PRs) by automerging changes that consist solely of known-safe dependency bumps. Named by analogy with evergreen browsers that silently self-update. Proposed as a stretch-goal supplementary workflow.
 
 ## F
 
+### Fail-Open
+
+When a step's error or a `fail` score must not fail the surrounding job or block delivery. Eval measurements are fail-open: a missing manifest, a scorer `fail`/`skip` label, or a measure-step IO error never fails the agent run. Contrast with fail-closed gates (auth, kill switch) where an error must stop the run. In scripts, fail-open is acceptable for non-critical steps (logging, metrics) and dangerous for gates.
+See [ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md), [Eval Measurements](guides/infrastructure/eval-measurements.md), and [Shell scripting](contributing/shell-scripting.md).
+
 ### Flapping
 
 When agents enter a cycle of conflicting feedback that prevents convergence. Example: the security review agent rejects what the code agent produces to satisfy the correctness review agent, and vice versa, creating an oscillating loop. Flapping is a primary trigger for [escalation](#escalation) — after a configurable number of cycles, the system stops and routes to humans.
@@ -147,6 +152,16 @@ See [security-threat-model.md](problems/security-threat-model.md).
 The logging, tracing, and audit layer for agent actions. Every agent action must be attributable, traceable, and reviewable — both for debugging failures and for security auditability. In practice, this includes capturing agent JSONL logs (including "thinking" traces), converting them to human-readable format, and uploading them as artifacts. Observability is a cross-cutting concern that touches every other component.
 See [architecture.md](architecture.md).
 
+### OTEL Derived Products
+
+Values **computed from** a run's OpenTelemetry trace after the fact — scores, fitness checks, later quality signals. They are not a second copy of what happened. First-ship example: `eval-measurements.jsonl` from `fullsend eval-measure` ([eval measurements](#eval-measurement) are the concept of scoring traces). Derived products sit beside telemetry as sibling files; they never replace [OTEL primary facts](#otel-primary-facts).
+See [ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md) and [Eval Measurements](guides/infrastructure/eval-measurements.md).
+
+### OTEL Primary Facts
+
+What **actually happened** on an agent run, recorded as OpenTelemetry (OTEL) spans. The local source of truth is `run-telemetry.jsonl`; when `OTEL_EXPORTER_OTLP_*` is set, the same spans also export live over OTLP ([ADR 0050](ADRs/0050-distributed-tracing-instrumentation.md)). Agent identity, work item, tokens, cost, span tree, and `exit_code` belong here. Sibling files (including [eval measurements](#eval-measurement)) must not become a second source of run truth.
+See [Distributed Tracing](guides/infrastructure/distributed-tracing.md) and [ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md).
+
 ## P
 
 ### Policy Store

docs/guides/README.md

@@ -27,6 +27,7 @@ Advanced guides for platform operators who deploy and manage the GCP-side infras
 - [Infrastructure reference](infrastructure/infrastructure-reference.md) — Token mint, WIF, and secrets deployment details
 - [Enabling fullsend on private repositories](infrastructure/private-repositories.md) — Additional guardrails and configuration for private repos
 - [Tracing reference](infrastructure/distributed-tracing.md) — Telemetry levels, environment variables, span hierarchy, and attributes
+- [Eval measurements](infrastructure/eval-measurements.md) — Online trace scoring with `eval-measurements.jsonl` and measurement manifests
 
 ## User guides