Skip to content

evals: capability-policy dataset for the exact capability layer - #1

Merged
th3nolo merged 3 commits into
th3nolo:mainfrom
semantic-debt:evals/capability-policy-dataset
Oct 15, 2026
Merged

th3nolo merged 3 commits into
th3nolo:mainfrom
semantic-debt:evals/capability-policy-dataset

Conversation

@semantic-debt

@semantic-debt semantic-debt commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

The existing frozen benchmark carries no trusted, predeclared exact capabilities, so it cannot measure the known-benign capability matcher. This adds an offline synthetic corpus and runner for that layer, together with fail-closed grant validity checks.

Behavior and scope

  • Add 1,080 deterministic cases in 90 families across ten primitives: 936 public cases and 144 held-out cases. Every family contains one authorized base action and eleven counterfactuals. Splitting by family keeps counterfactual siblings together.
  • Honor optional not_before and revoked_at fields. A grant that has not started, has been revoked, or has an invalid timestamp cannot match. UTC conversion outside Python's supported datetime range returns a non-match instead of raising an exception.
  • Preserve LF bytes for the public corpus through Git attributes, including checkouts with core.autocrlf=true, so its frozen SHA256 is portable across Linux and Windows.
  • Make the generator's --check validate the public file and both splits' manifest hashes, counts, label counts, paths, publication flags, and total family count. Verification regenerates in memory and never writes or scores held-out cases. Remove the always-true family-count assertion.
  • Keep dataset labels separate from authorization: grants are synthetic fixtures and never become runtime policy. A non-match falls through to the remaining gate layers.

The public corpus and manifest retain their original bytes and hashes. The documentation describes the stronger check command and the dataset's limits.

Regression coverage

  • Exercise Git's real checkout filtering with core.autocrlf=true and verify the pinned corpus hash.
  • Reject corrupt public/private hashes, counts, label counts, paths, publication flags, total family counts, and invalid manifest JSON. Check mode preserves the input manifest.
  • Reject UTC-overflow dates in both not_before and expires_at; retain correct inclusive start and exclusive expiry boundaries with non-UTC offsets.

The added regressions produced 19 failures before the fixes. The final focused suite passes 43 tests.

Validation

Validated commit: 47f5fd926416c45ca52e41c5d3e80da0769c4f7b, including the current main and the Windows scratch-path correction.

  • Local Windows / Python 3.13.7: python -m pytest -q tests — 3,280 passed, 12 skipped.
  • Full Linux/Windows matrix — all six jobs passed on Python 3.10, 3.12, and 3.13. Python 3.10: 3,279 passed, 13 skipped per OS; Python 3.12/3.13: 3,280 passed, 12 skipped per OS.
  • Normal PR CI — passed.
  • python -m pytest -q integrations/gemini-cli-shadow/tests -p no:cacheprovider — 54 passed, locally and in each matrix job.
  • python -m pip wheel . --no-deps -w wheelhouse — built successfully. Clean wheel-install tests with SEMGATE_WHEEL_DIR=wheelhouse — 2 passed, locally and in each matrix job.
  • Offline injection, trace-drift, and scripted chat-approval evaluations passed the workflow assertions, including zero automatic injection allows, zero false chat approvals, and zero chat code-path failures.
  • Public capability runner — 936 cases, 78/78 base cases matched, zero false allows, zero unnecessary asks.
  • Generator --check confirms the public corpus and both splits' manifest metadata match regeneration.
  • git diff --check origin/main HEAD passed.

All model evaluations in this slice use offline providers; there are no paid model calls.

The frozen benchmark carries no trusted, predeclared exact capabilities, so
it cannot measure the known-benign capability layer (the conclusion of
evals/3-known-benign-policy-layer.py). This adds the missing measurement:

- fixtures/eval/capability-policy.jsonl: 1080 frozen synthetic cases
  (936 public / 144 held out) in 90 families across ten primitives (file,
  repo, shell, network, message, deploy, permission, credential, money),
  reversible and irreversible. Each family pairs an exact in-scope base
  action with eleven minimally changed counterfactuals (wrong target,
  wider scope, expired, future, revoked, untrusted provenance, missing
  field, malformed, extra argument, wildcard, primitive-specific evasion).
  Split by family id; siblings never leak across splits.
- evals/27-gen-capability-policy.py: deterministic generator (fixed
  decision time); regeneration reproduces the committed file byte for byte.
- evals/run-capability-policy.py: offline runner, no model, no paid calls;
  exit 2 on any false allow or unmet base case. Public corpus: 0 false
  allows, benign auto-allow 78/78.
- semgate/capabilities.py: fail-closed validity window. The matcher now
  honors optional not_before and any non-null revoked_at; before this, a
  revoked or not-yet-valid grant still matched. Grants without those
  fields behave exactly as before.
- tests: matcher window tests plus set tests (regeneration, freeze rules,
  split isolation, clean score, labels never construct grants).

Dataset labels measure outcomes; they never create grants. Grants in the
set are synthetic fixtures, never runtime policy.
@semantic-debt
semantic-debt force-pushed the evals/capability-policy-dataset branch from 3fbcd7a to ca2cfe7 Compare September 30, 2026 00:05
@th3nolo
th3nolo merged commit ad49884 into th3nolo:main Oct 15, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants