Repository navigation
Largha and Decima 2.3 (agent 2.3, guard, guard-small), the decima-guard preset - #86
Merged
Merged
Conversation
…pins Follow decima-ai v2.3.0 (1b07634e): render.py choice_texts keeps an option's full text only when its token count fits; the 62-character shortcut is gone in the Rust port and the Python layout twin. The reference code moves to v2.3.0. Catalog: decima:agent moves to decima-agent v2.3 (652d90ac, 8,192-token states); decima:guard (98d3f734) and decima:guard-small (ac110248, mmBERT-small) are pinned at v1.0-preview. No manifests yet: export, goldens and parity come next (#80). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
check.long_cases: an agent transcript as the state, with a noul and a choice question, at about 3k, 6k and 8.1k mmBERT tokens (state rows of 3,041, 5,986 and 8,076 tokens on agent, kept whole) and 8.3k (8,266, upstream's STATE_TRUNCATED). small, base and the guards truncate all four. They are part of extra_cases, so goldens.py writes them too. Port vs upstream (v2.3.0), 483 identical requests, 0 mismatches: small 16 truncated, base 11, agent 1 (the 8.3k state). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
parity_run records only examples that print `record: {json}`; without it every
decima tag was reported as failed and nothing went into results/runs/. The line
carries the clef/decider keys (questions, decisions_same, max_logit_diff on the
raw option scores, max/p99 prob diff) plus decima's own gates: the ordinal
projections, the /v1/systemone answer diff and other choices.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Largha v1.1 (Phocinae/Phocinae-Largha-150M-v1 @ 5300e85) is a laya DecisionModel on mmBERT-small, so it goes through the laya export with its own repo, commit and root. laya_ref gains a root per checkpoint, `_laya` takes the repo, commit, root and export slug, and package.py writes aliases for laya-style tags too (largha:latest -> largha:150m). Parity against laya's own loader in fp32 on 477 cases (2,383 questions): identical ids and markers and every decision the same on the CPU (1.9e-6), CUDA (2.9e-6) and Metal through MLX (3.5e-6); the fp16 graph agrees on 2,380 of 2,383 (near-ties). Its typed-decisions score is marked trained-on-split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The questions of decima/presets/guard.json at 1b07634, unchanged: injection, harmful, toxic and the OWASP LLM Top 10 risk. guard-rag and guard-output read fields of a JSON state through `requires` and a state template, which Ollaya presets don't have, so they stay out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Against upstream v2.3.0 on 154 requests (599 questions) per model, CPU and CUDA (RTX 4090): identical rows and truncation, the same 18 rejections, every decision the same, scores within 3.2e-5 and probabilities within 8.2e-6. Agent 2.3's set has states of 3,041 to 8,266 tokens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench_plan's list from the candidate registry: largha:150m, decima:guard and decima:guard-small (new tags), decima:agent (new weights), and latency for decima:base and decima:small (parity- identical graphs). Public macro accuracy: agent 2.3 0.534, guard 0.518, guard-small 0.424, largha 0.398; largha's typed-decisions row is marked trained on the split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The models for 0.15.0, batched. Closes #71 and #80.
Largha (#71). Phocinae's Largha v1.1 (
Phocinae/Phocinae-Largha-150M-v1@5300e85) is a layaDecisionModelon mmBERT-small, so it goes through the laya export with its own repo, commit and root:laya_refgets a root per checkpoint, and_layatakes the repo, commit, root and export slug.package.pywrites aliases for laya-style tags too (largha:latest→largha:150m).bench_planleaves out its single-precision tags, as it does for laya's.Parity against laya's own loader in fp32, 477 cases (2,383 questions): identical ids and markers and every decision the same on the CPU (1.9e-6), CUDA (2.9e-6) and Metal through MLX (3.5e-6). The fp16 graph agrees on 2,380 of 2,383 decisions; the other 3 are near-ties.
Decima 2.3 (#80).
render.pyatv2.3.0does. One request changes,decima/fit/noul_few_chars.decima:agentmoves to 2.3 (652d90a), with 8,192-token states.decima:guardanddecima:guard-small, both 1.0 preview. guard-small is the first mmBERT-small Decima.check.pyhas long agent states (3,041 to 8,266 tokens), andparity_decimaprints a record line forparity_run. Against upstreamv2.3.0on 154 requests (599 questions) per model, CPU and CUDA (RTX 4090): identical rows and truncation, the same 18 rejections, every decision the same, scores within 3.2e-5 and probabilities within 8.2e-6.Preset.
decima-guardadds the questions of Decima'sguardpreset atv2.3.0as a built-in.guard-ragandguard-outputstay out: they read fields of a JSON state throughrequiresand a state template, which Ollaya presets don't have.Bench (edition 2026.10, update 0.15.0). These are the rows
bench_planlisted, measured from the candidate registry on the RTX 5090:decima:base and decima:small got latency only, because their graphs are parity-identical.
Checks.
cargo fmt --check, clippy-D warningsandcargo test --workspacepass (285 tests).python3 -m unittest discover -s convert/testspasses.npm run buildpasses, also with the candidate manifests inregistry/: the new pages render their result references.The registry manifests are not in this PR; they go in at release time.
🤖 Generated with Claude Code