Repository navigation
evals: capability-policy dataset for the exact capability layer - #1
Merged
th3nolo merged 3 commits intoOct 15, 2026
Merged
Conversation
The frozen benchmark carries no trusted, predeclared exact capabilities, so it cannot measure the known-benign capability layer (the conclusion of evals/3-known-benign-policy-layer.py). This adds the missing measurement: - fixtures/eval/capability-policy.jsonl: 1080 frozen synthetic cases (936 public / 144 held out) in 90 families across ten primitives (file, repo, shell, network, message, deploy, permission, credential, money), reversible and irreversible. Each family pairs an exact in-scope base action with eleven minimally changed counterfactuals (wrong target, wider scope, expired, future, revoked, untrusted provenance, missing field, malformed, extra argument, wildcard, primitive-specific evasion). Split by family id; siblings never leak across splits. - evals/27-gen-capability-policy.py: deterministic generator (fixed decision time); regeneration reproduces the committed file byte for byte. - evals/run-capability-policy.py: offline runner, no model, no paid calls; exit 2 on any false allow or unmet base case. Public corpus: 0 false allows, benign auto-allow 78/78. - semgate/capabilities.py: fail-closed validity window. The matcher now honors optional not_before and any non-null revoked_at; before this, a revoked or not-yet-valid grant still matched. Grants without those fields behave exactly as before. - tests: matcher window tests plus set tests (regeneration, freeze rules, split isolation, clean score, labels never construct grants). Dataset labels measure outcomes; they never create grants. Grants in the set are synthetic fixtures, never runtime policy.
semantic-debt
force-pushed
the
evals/capability-policy-dataset
branch
from
September 30, 2026 00:05
3fbcd7a to
ca2cfe7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The existing frozen benchmark carries no trusted, predeclared exact capabilities, so it cannot measure the known-benign capability matcher. This adds an offline synthetic corpus and runner for that layer, together with fail-closed grant validity checks.
Behavior and scope
not_beforeandrevoked_atfields. A grant that has not started, has been revoked, or has an invalid timestamp cannot match. UTC conversion outside Python's supported datetime range returns a non-match instead of raising an exception.core.autocrlf=true, so its frozen SHA256 is portable across Linux and Windows.--checkvalidate the public file and both splits' manifest hashes, counts, label counts, paths, publication flags, and total family count. Verification regenerates in memory and never writes or scores held-out cases. Remove the always-true family-count assertion.The public corpus and manifest retain their original bytes and hashes. The documentation describes the stronger check command and the dataset's limits.
Regression coverage
core.autocrlf=trueand verify the pinned corpus hash.not_beforeandexpires_at; retain correct inclusive start and exclusive expiry boundaries with non-UTC offsets.The added regressions produced 19 failures before the fixes. The final focused suite passes 43 tests.
Validation
Validated commit:
47f5fd926416c45ca52e41c5d3e80da0769c4f7b, including the currentmainand the Windows scratch-path correction.python -m pytest -q tests— 3,280 passed, 12 skipped.python -m pytest -q integrations/gemini-cli-shadow/tests -p no:cacheprovider— 54 passed, locally and in each matrix job.python -m pip wheel . --no-deps -w wheelhouse— built successfully. Clean wheel-install tests withSEMGATE_WHEEL_DIR=wheelhouse— 2 passed, locally and in each matrix job.--checkconfirms the public corpus and both splits' manifest metadata match regeneration.git diff --check origin/main HEADpassed.All model evaluations in this slice use offline providers; there are no paid model calls.