feat(intrinsics): one-token decisions (decide/choose/score)

[planetf1]

Pull Request

Issue

Fixes #1741

Description

Granite Switch models can now answer yes/no questions, pick from options, and rate severity, in a single token. You ask the question in plain English, get back a number you can if on, and it costs one forward pass instead of generating a paragraph to parse.

from mellea.stdlib.components.intrinsic import decisions

# Is this urgent?
if decisions.decide(ticket, backend, "Does this require urgent attention?") >= 0.7:
    escalate(ticket)

# Which team should handle this?
route = decisions.choose(ticket, backend, "Which team handles this?",
                         {"payments": "Payout failures and refunds",
                          "account": "Login and account access"})
# route.choice → "payments"

# How bad is it, 0-2?
sev = decisions.score(ticket, backend, "How urgent is this?",
                      ["can wait a week", "this week", "today"])
# sev.score → 1.89

The state argument (first parameter) is what you're asking about: a string, a dict, or a whole conversation. For background on the updated Granite Switch 4.1 release, including the decision capability this PR adds support for, see the announcement blog post. All three helpers take model_options for temperature (it has no effect on the returned answer, only on the sampled token shown in the log); everything else that matters for the readout is pinned and will raise if you try to override it.

How to serve it: vLLM with --max-logprobs 20 and the granite-switch package. Also works with llama.cpp's llama-server. See the adapter-functions docs for full setup, and runnable examples in docs/examples/intrinsics/decisions.py and decisions_llamacpp.py.

Supporting changes to get the latest Granite Switch code working:

• Bumped granite-switch[hf] from >=0.1.0 to >=0.1.1. The 0.1.1 release carries the multiswitch layer-accounting fix that the new-generation checkpoints need; without it, loading them on the HuggingFace path produces garbage output.
• Dropped the override-dependencies transformers pin from [tool.uv]. It was there to widen 0.1.0's <5.10.0 restriction, which no longer applies; 0.1.1's own pin (>=5.16) is within Mellea's range, so dev environments now resolve the same transformers version as downstream users.

Hardened embedded-adapter discovery:

• Field-by-field config merge: a partially published io.yaml no longer breaks calls
• Merged-config validation: invalid configs disable decision with the reason in the caller's error
• Chat-template invocation check: mismatch warns (does not disable)
• adapter_revision parameter on both backends to pin discovery to the server's commit

Documentation and examples:

• A step-by-step tutorial (Tutorial: One-Token Decisions) covering a realistic ticket-triage workflow, customizing true/false descriptions, using a conversation as state, reading log entries, and handling all three failure modes
• An expanded section on the adapter-functions page with serving setup, API reference, how the readout works, answer semantics, and failure modes
• Two runnable examples: decisions.py (vLLM) and decisions_llamacpp.py (llama.cpp), both linked from the examples index

What's verified vs what's not:

The client-side machinery is fully verified: prompt layout is byte-identical to the reference, the readout works on vLLM/llama.cpp/HuggingFace, error handling is tested, and answers agree across backends. An independent cross-check against the reference decide_alora.py script (running the published aLoRA on the base 8B model) confirmed that mellea's probabilities match the reference to within 0.004 over six test cases, with the same top answer every time, which also settles numerical parity with the PEFT path. The e2e battery now includes diverse questions (factual reasoning, subjective judgment, routing, severity, intent categorization, JSON state) alongside the original sanity tests.

What is not proven here, and needs a separate benchmark:

• Accuracy on a labeled test set. The diverse-question tests check that answers are sensible, not that they are measurably correct.
• Calibration. Whether P(urgent) = 0.95 actually means 95% correct is unmeasured.
• Adapter value-add. The base model (without the decision adapter) also concentrates on option labels. How much the adapter improves accuracy over the base model is unknown.

These are model-quality questions, not client-correctness questions. They need a labeled evaluation set and are the model team's domain.

What's not in this PR (tracked in #1741):

• Readout failures currently map to the generic error telemetry outcome rather than schema_error; fixing this needs a hook payload extension.

Testing

• Tests added to the respective file if code was changed

• New code has 100% coverage if code was added

• Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

• 415+ unit tests covering rendering rules, merge semantics, answer types, error classes, and backend-default protection

• Golden-layout parity test against the reference prompt construction (8 tests, env-gated on staged artifacts)

• Env-gated e2e battery (21 tests: activation probe, reference smoke, sanity, diverse questions, readout mechanics, latency, prefix cache) verified live on vLLM (20 passed, 1 skipped); falls back to the nightly script's env vars

• HuggingFace backend verified with granite-switch 0.1.1 + transformers 5.17.0

• Full suite: 5085 passed, 1 pre-existing failure (upstream pin drift, CI-skipped)

• Gates: ruff, mypy, docs-autogen 100%/0, all clean

Attribution

• AI coding assistants used

---

Adding a new component, requirement, sampling strategy, or tool?

If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.

• Component
• Requirement
• Sampling Strategy
• Tool

NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing.

[jakelorocco]

Does this PR rely on the other open llama server PR?

[jakelorocco]

Does this need to take the revision for the embedded adapter?

[jakelorocco]

Claude is saying that temperature does impact the LocalHFBackend since "probabilities are built from scores taken after temperature and top_k/top_p are applied"?

[jakelorocco]

This also needs to check any model options set at the backend level right?

[jakelorocco]

I think this merging is shallow and breaks if extra_body is a dict that is set. The new extra_body will replace every other option in the original dict.

[jakelorocco]

Should we still handle other contexts here? and convert them into strings properly?

[jakelorocco]

If we are only supporting one name now, there's a bunch of references in the code about multiple decision capability names.

[planetf1]

My error in leaving in - the name changed between earlier development and later checkpoints. I removed the other values but not fully the mechanism.

[planetf1]

Verification: decide/choose/score on both GA Switch checkpoints

Ran the decision e2e battery plus a guardian-core probe against ibm-granite/granite-switch-4.1-8b and ibm-granite/granite-switch-4.1-30b (the GA checkpoints embedding guardian-core + decision) on all three supported serving paths. Everything passes.

Path	8b	30b
llama.cpp llama-server (GGUF via the pinned converter, see #1736)	19 passed, 2 skipped	19 passed, 2 skipped
vLLM with --max-logprobs 20	20 passed, 1 skipped	20 passed, 1 skipped
HF transformers, LocalHFBackend with embedded adapters	pass	pass

The skips are deliberate: the latency comparison needs a checkpoint that embeds requirement-check, which the GA models do not carry, and the prefix-cache assertion reads vLLM's /metrics.

On the reference case (the payouts ticket), the 8b gives P(urgent) 0.95 on llama.cpp and vLLM and 0.99 on HF, routes to payments at 0.98 or better, scores severity 1.89 of 2, and guardian-core flags a self-harm prompt at 0.84 against about 1e-5 for a benign one. The 30b lands close to itself on every path: P(urgent) 0.92, payments 0.99, severity 1.5, guardian 0.99 against about 1e-4. That kind of cross-path agreement (within ~0.005 on the label distributions) is also a fresh confirmation that the GGUF conversion is faithful. Both examples (decisions.py, decisions_llamacpp.py) run clean on their respective paths.

The earlier build-and-validate failure was my fault: the Returns section of FirstTokenChoice.apply used <sampled top-1 token> style placeholders, which the docs generator wraps into the API reference MDX where Docusaurus reads them as JSX tags. Reworded in 6d24522; autogen, the docstring quality gate, and a full Docusaurus build pass locally, and the check is green again.

A full-suite regression run against this branch is still in progress. I will follow up with the summary once it completes.

[planetf1]

Follow-up as promised: the full-suite regression run against this branch finished. On Python 3.11, 3.12 and 3.13 the suite passes (5068 / 5068 / 5069) with two failures per version, both pre-existing on main (test_adapter_versions_unchanged, the granitelib pin-drift check, and test_message_typing). The Granite Switch intrinsic suite, 55 tests per version against the preview checkpoint that embeds the twelve catalog adapter functions, is fully green. That covers the formatter and output-processor changes this PR makes on the shared intrinsic path, on top of the decision-specific battery above.