Repository navigation
Commit 4caadd5
feat(runner-watch): structured gate, value normalizers, coverage map, tree-sitter deltas (#208)
* feat(runner-watch): structured gate, value normalizers, coverage map, tree-sitter deltas
Addresses four conformance-tooling gaps.
#1 Structured verdict instead of scraping the markdown report.
- compare::analyze() -> ConformanceReport with failures(&GatePolicy) + to_json();
conform() gates on the struct and writes conformance-result.json. render_report
is now a human-only artifact, no longer parsed for pass/fail. Removed the four
string-scraping gate fns and their fixture tests; gate semantics preserved
(status-ignore, request-schema-any-diff, acquirejob response-field-drop via a
structural schema_drops_fields).
#4 Declarative value normalizers + optional value gating.
- normalize_value() collapses volatile scalars (GUID/JWT/token/URL/timestamp/
big-int/high-entropy) while keeping stable values visible. New ValueGate policy;
default Off preserves legacy pass/fail and only reports value diffs.
--value-gate-strict opts into normalized response-value gating for all shared
endpoints except known-volatile ones. Strict is opt-in because validating its
false-positive rate needs a live golden replay.
#2 Endpoint coverage map (new coverage.rs + `runner-watch coverage`).
- Cross-references golden-exercised endpoints against routes scraped from the
server router, restricted to the runner-facing protocol surface. Warn by
default; --strict fails on uncovered runner-facing routes minus
.runner-watch/coverage-allow.txt. Against v2.336.0: 18 covered, 103 uncovered
alias routes, 3 genuine gaps (runner calls /_apis/distributedtask/pools etc.
that the server serves only under /runner/server or /_apis/v1).
#3 tree-sitter-c-sharp delta extraction (was line heuristics).
- Detects fields, record parameters, enum members, methods, expression-bodied
properties; heuristic fallback retained. DeltaEntry/change_type unchanged.
Verification: cargo test -p runner-watch (47 passing incl. gate-the-gate and
coverage-over-real-goldens), fmt, workspace clippy -D warnings.
Entire-Checkpoint: 01M1CS16K18E9CMAQWHQREYS0X
* parser: serialize job env as TemplateToken maps for the official runner
The official runner deserializes AgentJobRequestMessage
.EnvironmentVariables into IList<TemplateToken> with a schema-driven
evaluator. A plain JSON object per variable (the previous wire shape)
converts to a token the workflow schema rejects, failing every job that
declares job-level env with "The template is not valid. Unexpected
value ''". Emit the {type: 2, map: [{Key, Value}]} shape GitHub sends
(values are already resolved, so literal type-0 scalars are correct).
Found via pl-official conformance captures: every job-level env
workflow failed on the official runner against preloop while passing on
GitHub.
Entire-Checkpoint: 01M1CVAAMRC4D8H66JVAKBPBNV
* fix(runner-watch): address PR review on coverage, value gate, and C# deltas
- coverage: unify path normalization across both sides. `canonicalize` now
reuses `compare::strip_transport_prefixes` (shared with `normalize_path`) and
drops a leading org/base segment before `/_apis`, so `/runner/server/…` and
`/{org}/_apis/…` aliases match the goldens instead of showing as false gaps.
Real corpus: covered 18->22, uncovered 103->67, false gaps 3->1. (Codex A1 /
CodeRabbit prompt)
- coverage: preserve HTTP method — routes and golden keys are now `METHOD path`,
so GET+POST on one path aren't conflated. Route methods parsed from the
handler chain (paren-aware), `*` when none found. (Codex A2)
- coverage: catch-all (`*wild`) routes match by prefix instead of exact
equality, so multi-segment golden paths under `/replay/results/*path` count as
covered. (CodeRabbit major)
- coverage: `coverage_cmd` prints the allowlist-filtered uncovered count, not the
raw one. (CodeRabbit minor)
- compare: `normalize_value` collapses large integer JSON numbers (>=1e9:
runner ids, epochs) so `--value-gate-strict` doesn't false-positive on numeric
ids; small numbers stay intact. (CodeRabbit + Codex)
- main: C# methods are keyed by name+parameter types, so an added/removed
overload produces a distinct delta instead of collapsing on the `(type, name)`
map. (Codex)
- docstrings added on touched items.
Tests: 50 passing (adds numeric-normalize, method-overload, and method/alias/
catch-all coverage tests). fmt + clippy -D warnings clean.
Entire-Checkpoint: 01M1D2B39HZ2PJVEF224QS9K0E
* fix(runner-watch,parser): address second-round PR review
runner-watch/compare.rs
- body_comparison compares every occurrence in an endpoint group by index, not
just the first, so a later call that regresses its body is caught (a deduped
array would have hidden a per-occurrence field drop). (CodeRabbit major)
runner-watch/main.rs
- method_signature now encodes generic arity (`M<2>`) and each parameter's
passing mode (`ref int`), so ref/out/in and generic-arity overloads produce
distinct keys instead of overwriting each other. (Codex/CodeRabbit)
- parse_csharp returns None when the tree has syntax errors, so extract_fields
falls back to the line heuristic instead of using an incomplete model.
(CodeRabbit)
runner-watch/coverage.rs
- covers() treats a root catch-all ("/") as matching every descendant instead
of building a "//" prefix. (CodeRabbit)
parser/job_builder.rs
- job env values serialize through template_string_token, so runtime-only
expressions (e.g. `${{ github.workspace }}/bin`) reach the runner as
expression tokens rather than raw literals; plain values stay type-0 literals.
(CodeRabbit major)
- tests assert the TemplateToken wire shape (type 2 map, literal Key/Value) and
add a github.workspace regression.
Tests: runner-watch 54, gha-parser job_env 2/2; fmt + clippy -D warnings clean
on both crates. (Unrelated uncommitted runs.rs WIP in the checkout left
untouched.)
Entire-Checkpoint: 01M1D6JSFYFCVW1SM7ZYNRHW9D
* fix(runner-watch): address cubic review
compare.rs
- body_comparison: treat a `null` or absent body as "no body" so an omitted
official body vs a replayed `null` is not a false RequestSchema mismatch (P1
blocker); align bodies by flow slot (Option per flow) instead of filtering,
so occurrences at different indexes aren't silently re-aligned; compare only
the shared min-count of occurrences so a cardinality difference (extra poll /
skipped retry) isn't folded into the body gate.
coverage.rs
- add /api/v1/actions to runner-facing prefixes so the bearerless action-download
route's coverage is tracked.
- restrict the leading-segment strip before /_apis to the `{p}` (org param)
case, so a literal base like /foo_bar/_apis/... is no longer collapsed onto
the bare /_apis route.
- render_markdown: "0/0 (n/a)" for an empty route set (not 100%), and trim a
leading `v` so a v-prefixed version doesn't render "vv2.336.0".
main.rs
- method_signature builds each parameter from the `type` field + `modifier`
nodes rather than slicing raw text before the name, so a default value that
contains the parameter name (`string s = "s"`) no longer truncates the
signature.
- coverage_cmd reads the allowlist before writing the JSON artifact and adds an
`uncovered_after_allowlist` field so the JSON matches the printed/strict
verdict.
- read_coverage_allowlist propagates non-NotFound read errors instead of
silently treating an unreadable allowlist as empty.
Deliberately not changed: the >=1e9 numeric normalization in normalize_value.
Cubic flags it as possibly hiding a stable large number, but it was explicitly
requested last round (Codex/CodeRabbit) to stop false ValueMismatch on numeric
ids/epochs, only affects opt-in --value-gate-strict, and there is no clean
value-only heuristic distinguishing an id from a large count. Left as-is.
Tests: runner-watch 56, gha-parser 139; fmt + clippy -D warnings clean.
Entire-Checkpoint: 01M1D8JTW5DT9F2BXJXZXXQVS4
---------
Co-authored-by: bnjoroge <bnjoroge@local>1 parent e8c38b0 commit 4caadd5
6 files changed
Lines changed: 1935 additions & 223 deletions
File tree
- crates
- preloop-gha-parser/src
- runner-watch
- src
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
458 | 458 | | |
459 | 459 | | |
460 | 460 | | |
461 | | - | |
462 | | - | |
463 | | - | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
464 | 472 | | |
465 | 473 | | |
466 | | - | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
467 | 483 | | |
468 | 484 | | |
469 | 485 | | |
| |||
1020 | 1036 | | |
1021 | 1037 | | |
1022 | 1038 | | |
| 1039 | + | |
| 1040 | + | |
| 1041 | + | |
| 1042 | + | |
| 1043 | + | |
| 1044 | + | |
| 1045 | + | |
| 1046 | + | |
| 1047 | + | |
| 1048 | + | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
| 1052 | + | |
| 1053 | + | |
| 1054 | + | |
| 1055 | + | |
| 1056 | + | |
| 1057 | + | |
| 1058 | + | |
| 1059 | + | |
| 1060 | + | |
| 1061 | + | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
| 1065 | + | |
| 1066 | + | |
| 1067 | + | |
| 1068 | + | |
1023 | 1069 | | |
1024 | 1070 | | |
1025 | 1071 | | |
| |||
1050 | 1096 | | |
1051 | 1097 | | |
1052 | 1098 | | |
1053 | | - | |
1054 | | - | |
1055 | | - | |
1056 | | - | |
1057 | | - | |
1058 | | - | |
| 1099 | + | |
1059 | 1100 | | |
1060 | 1101 | | |
1061 | 1102 | | |
| 1103 | + | |
| 1104 | + | |
| 1105 | + | |
| 1106 | + | |
| 1107 | + | |
| 1108 | + | |
| 1109 | + | |
| 1110 | + | |
| 1111 | + | |
| 1112 | + | |
| 1113 | + | |
| 1114 | + | |
| 1115 | + | |
| 1116 | + | |
| 1117 | + | |
| 1118 | + | |
| 1119 | + | |
| 1120 | + | |
| 1121 | + | |
| 1122 | + | |
| 1123 | + | |
| 1124 | + | |
1062 | 1125 | | |
1063 | 1126 | | |
1064 | 1127 | | |
| |||
1107 | 1170 | | |
1108 | 1171 | | |
1109 | 1172 | | |
1110 | | - | |
1111 | | - | |
1112 | | - | |
1113 | | - | |
1114 | | - | |
1115 | | - | |
| 1173 | + | |
1116 | 1174 | | |
1117 | 1175 | | |
1118 | 1176 | | |
| |||
1122 | 1180 | | |
1123 | 1181 | | |
1124 | 1182 | | |
| 1183 | + | |
| 1184 | + | |
| 1185 | + | |
| 1186 | + | |
| 1187 | + | |
| 1188 | + | |
| 1189 | + | |
| 1190 | + | |
| 1191 | + | |
| 1192 | + | |
| 1193 | + | |
| 1194 | + | |
| 1195 | + | |
| 1196 | + | |
| 1197 | + | |
| 1198 | + | |
| 1199 | + | |
| 1200 | + | |
| 1201 | + | |
| 1202 | + | |
| 1203 | + | |
| 1204 | + | |
| 1205 | + | |
| 1206 | + | |
| 1207 | + | |
| 1208 | + | |
| 1209 | + | |
| 1210 | + | |
| 1211 | + | |
| 1212 | + | |
| 1213 | + | |
| 1214 | + | |
| 1215 | + | |
| 1216 | + | |
| 1217 | + | |
| 1218 | + | |
| 1219 | + | |
| 1220 | + | |
| 1221 | + | |
| 1222 | + | |
| 1223 | + | |
| 1224 | + | |
| 1225 | + | |
| 1226 | + | |
| 1227 | + | |
| 1228 | + | |
| 1229 | + | |
| 1230 | + | |
| 1231 | + | |
| 1232 | + | |
| 1233 | + | |
| 1234 | + | |
| 1235 | + | |
| 1236 | + | |
| 1237 | + | |
| 1238 | + | |
| 1239 | + | |
1125 | 1240 | | |
1126 | 1241 | | |
1127 | 1242 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
| 26 | + | |
25 | 27 | | |
0 commit comments