SystemOne Decision Models Integration into lemonade

[meghsat]

Proposal size

Major feature (spans multiple well-scoped PRs)

Updates

Implemented this here: #3752

User story

As a developer I want to get decisive, straightforward, and quick responses to my questions and LLMs[system-two] models are slow and often produce way too many tokens unless we prompt tune them. To solve this, a new set of models called system-one is released. These models are built to make fast, structured decisions that software can use directly.

llama.cpp added decision models in PR #29818. The PR was merged on 2026-10-02, and release b11361 contains it. A decision model answers typed questions in one forward pass. It does not generate tokens. llama-server serves these models through the /v1/systemone endpoint. The GGUF files are released for the following models:

llama-server -hf ggml-org/OpenJev-GGUF
llama-server -hf ggml-org/lev-GGUF
llama-server -hf ggml-org/Laya-GGUF
llama-server -hf ggml-org/Julia-1-GGUF
llama-server -hf ggml-org/Kev-4B-GGUF

These models can also be extended into the router colelction.router policy. As a developer, I want to ask a small local model typed questions about each request. Then I can route a request on rules that I write in plain words. For example: "Does the message contain personal data?" or "Which team must handle this?".

High-level design

Lemonade gives decision models to clients and to the router in three ways.

1. A new endpoint: POST /v1/systemone

The endpoint is also available under /api/v0/, /api/v1/, /v1/, and /v0/. Lemonade does these steps:

• It sends the request body to llama-server without changes. llama-server builds the prompt from the template in the GGUF file. Lemonade does not build prompts.
• It replaces the model field of the response with the Lemonade model name. llama-server writes the local file path into that field.
• Lemonade returns 400 before a model load if the request has no state, has no questions, or names a model that is not a decision model.

Request:

{
  "model": "Laya-GGUF",
  "state": "Customer message: I was charged twice for my order last week and nobody has replied.",
  "questions": {
    "route":   {"type": "choice", "instructions": "Which team should handle this?",
                "criteria": {"billing": null, "shipping": null, "technical": null}},
    "angry":   {"type": "noul",   "instructions": "Is the customer angry?"},
    "urgency": {"type": "score",  "instructions": "How urgent is this?",
                "criteria": ["can wait", "this week", "today", "right now"]}
  }
}

Response (Laya, values rounded):

{
  "model": "Laya-GGUF",
  "answers": {
    "route":   {"type": "choice", "choice": "billing",
                "probabilities": {"billing": 0.9905, "shipping": 0.0066, "technical": 0.0029}, "confidence": 0.9858},
    "angry":   {"type": "noul", "noul": 0.7844},
    "urgency": {"type": "score", "score": 1.916,
                "legend": {"0": "can wait", "1": "this week", "2": "today", "3": "right now"},
                "probabilities": {"0": 0.0405, "1": 0.4560, "2": 0.0503, "3": 0.4531}, "confidence": 0.0029}
  },
  "usage": {"input_tokens": 136, "output_tokens": 0}
}

2. Zero-shot labels on POST /v1/classify

A decision model also answers a /v1/classify request that has labels. Lemonade asks the model one choice question, with the labels as the options. The response has the same form as the response of the ONNX zero-shot model. Thus the zero_shot router classifier from PR #3740 operates with a GGUF decision model, and the router does not change.

The recipe of the model selects the backend:

flowchart LR
    C1[Client] -->|POST /v1/systemone| H1[handle_systemone]
    P[collection.router policy] -->|systemone| RS[Router::systemone]
    H1 --> RS
    P -->|zero_shot| RC[Router::classify]
    C2[Client] -->|POST /v1/classify + labels| RC
    RC -->|onnxruntime| O[ort-server]
    RC -->|llamacpp| A[LlamaCppServer::classify<br/>one choice question]
    RS --> L[LlamaCppServer::systemone]
    A --> LS[llama-server /v1/systemone]
    L --> LS

Loading

A systemone classifier asks one typed question in your words. Its labels are the options of the question:

Question type	Labels
choice	The keys of criteria
noul	true and false
score	The level strings of criteria

{
  "classifiers": [
    {"id": "pii", "type": "systemone", "model": "Laya-GGUF", "default_label": "true",
     "question": {"type": "noul", "instructions": "Does the message contain personal data?"}},
    {"id": "topic", "type": "zero_shot", "model": "Laya-GGUF", "labels": ["chit-chat", "coding"]}
  ],
  "rules": [
    {"id": "pii-local", "match": {"classifier": "pii", "min_score": 0.5}, "route_to": "Local-GGUF"},
    {"id": "code-to-big", "match": {"classifier": "topic", "label": "coding", "min_score": 0.5},
     "route_to": "Big-GGUF"}
  ]
}

Lemonade adds these models to the registry:

Model	Decision type	Base	File	Size
Julia-1-GGUF	laya (encoder)	mmBERT small, 144M	Q8_0	0.17 GB
Laya-GGUF	laya (encoder)	ModernBERT, 421M	Q8_0	0.45 GB
Kev-4B-GGUF	kev (causal)	Qwen3.5 4B Base	Q8_0	4.48 GB
lev-GGUF	lev (causal)	Qwen3.5 4B	Q8_0	4.48 GB
OpenJev-GGUF	openjev (causal, images)	Qwen3.5 27B	Q4_K_M + mmproj	19.6 GB

The classification label is the deployment mode. Thus a decision model never loads as a chat model, and it does not unload the chat model of the user. The systemone label tells Lemonade that the model is a decision model. A llama.cpp model with the classification label and without the systemone label is still refused at registration.

Implementation

PR	Status	Content
ggml-org/llama.cpp#29818	Merged 2026-10-02	/v1/systemone in llama-server
#3740	Open	zero_shot classifier type, labels on /v1/classify
lemonade-sdk/ort-server#4	Open	Zero-shot task for the ONNX router model
lemonade-sdk/lemonade#____	To open	/v1/systemone, the /v1/classify adapter, the systemone classifier type, the launch settings, the two models, tests, and the spec docs/dev/specs/systemone-decision-models.md

Breaking changes

None. The endpoint, the classifier type, and the two models are additions.

Maintenance plan

CI tests the feature. Upstream llama.cpp maintains the prompt layout and the scores of each decision model.

Part	Covered by
/v1/systemone, /v1/classify adapter, systemone and zero_shot classifiers	CI job systemone-llamacpp (cpu, Julia-1 and Laya) and C++ unit tests
Prompt layout, scores, temperatures	Upstream llama.cpp
New decision models or types from upstream	We add a registry entry and a test

Risks

No response