llama.cpp now serves decision models through /v1/systemone

llama.cpp, a widely used engine for running models locally, can now run decision models. On October 2 the project merged pull request 29818, and its maintainers explained the feature in a Hugging Face article. A decision model scores the options you give it in one forward pass instead of generating text. This article covers what was added, which models work, a quick start, and a result relevant to Turkish. We read the announcement, the pull request, the server documentation and the model cards; we did not run any of it.

What was added

The llama.cpp server gains a /v1/systemone endpoint. You send a state (text, JSON or, for some models, an image) and typed questions, and the model returns a probability for each option. The API follows the System One format introduced with TypeSafe’s Jev, so, in the announcement’s words, existing clients only need a new base URL. The documentation lists the limits: streaming is not supported, an invalid request returns 400, and a model that is not a decision model returns 501.

There are three question types:

Type	You send	You get
choice	options, with optional descriptions	the top option and a probability per option
score	2 to 10 levels, lowest first	the expected level, which can fall between two
noul	a yes/no question	the probability of yes

The pull request describes decision models as wrappers around traditional embedding models, so the changes to the core library are small: a decision head on the graph, conversion support and a server context that switches input and output handling by model metadata. It also adds a shared prompt prefix for causal models, vision input and tests. The author’s disclosure in the pull request reads that most of the code is AI-written and that they own the design.

Which models work

Model	Size, base	Languages	License	Speed*
Julia-1	144M, mmBERT-small	50+	Apache-2.0	3 ms
Laya	421M, ModernBERT-large	English	Apache-2.0	5 ms
Kev-4B	4B, Qwen3.5-4B-Base	English	Apache-2.0	12 ms
lev	4B, Qwen3.5-4B	English	Apache-2.0	36 ms
OpenJev	27B, Qwen3.8-27B	en, de, fr, hi, zh, ja; also reads images	CC BY-NC 4.0	43 ms

*Median time to answer one question on one NVIDIA RTX PRO 6000, as ggml-org reports it. We did not measure these.

OpenJev’s non-commercial license is the one to check before using it in a product; the GGUF cards for the other four list Apache-2.0. ggml-org says more models are coming, and the pull request lists Cloudflare’s Clef as a planned follow-up. A community leaderboard, the Decision Index, compares the models.

Quick start

The announcement says to get the latest llama.cpp from llama.app (or run llama update) and start a model:

llama serve -hf ggml-org/Kev-4B-GGUF

A request carries one state and any number of questions:

curl http://localhost:8080/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "state": "Customer message: I was charged twice for my order last week and nobody has replied.",
    "questions": {
      "route": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {
          "billing": "payments, charges, refunds, invoices",
          "shipping": "delivery, tracking, lost or late parcels",
          "technical": "bugs, errors, login problems"
        }
      },
      "angry": { "type": "noul", "instructions": "Is the customer angry?" }
    }
  }'

In the announcement’s example run, with Kev-4B, the routing question returned billing at 0.905 and the anger question 0.82. In router mode, llama serve loads models on demand and you choose one per request with the model field; image input uses OpenJev, whose vision projector downloads automatically.

How closely it matches the originals

The pull request includes a test on one customer-support input with five questions, comparing each model’s reference output with llama.cpp’s. The worst difference in any probability:

Model	Largest difference
OpenJev	0.00018
Laya	0.00084
Kev-4B	0.0013
lev	0.0052
Julia-1	0.022

Julia-1’s gap is the largest and the only one above 0.01; the author did not run Julia-1 on one of the five questions. The test is a single input, so it shows parity on that case, not across tasks.

A Turkish result

Four of the five models are listed for English or a few other languages, and none of those lists includes Turkish. Julia-1, from Supersonic Labs, starts from the multilingual mmBERT-small encoder, and its published results file includes Turkish.

Julia-1 result (Supersonic Labs’ own evaluation)	Score
MASSIVE scenario classification, Turkish (tr-TR)	80.3% (2,388 of 2,974)
Same, English (en-US)	86.75%
Same, Portuguese (pt-PT)	86.25%
Same, all 52 locales (macro)	71.5%
Typed decisions, 2,000 items	73.15%, against 72.70% for a Jev reference figure
Banking77 pilot, 72 labels, 100 examples	64%, against 87% for the Jev reference

Read it with the card’s own caveats. MASSIVE measures 18 scenario labels, not intent classification or slot filling. The classification pilots use 100 examples each, the Jev numbers are supplied reference figures and not a new Jev run, and the card says Julia compares the answers you provide and is not established for supplying missing facts or multi-step reasoning. A native call accepts 2 to 20 options. The figures were measured by the model’s makers on September 24.

Tips from the announcement

• Describe your options. Julia-1 routed “I was charged twice” to shipping with bare labels and to billing (0.99) once each option had a description.
• Pick a confidence cutoff per model. A vague ticket scored 0.25 with Julia-1 and 0.80 with Kev-4B, so test on your own examples.
• Batch your questions. They are answered independently, and Kev-4B, lev and OpenJev process the state only once.
• Try other quantizations, for example ggml-org/Kev-4B-GGUF:Q8_0.

The server documentation adds that the probabilities are scaled with temperatures stored in the model file and are not guaranteed to be calibrated for your data.

What to keep in mind

These are our observations:

• It runs locally. Running a decision model in your own llama.cpp server keeps the text on your machine, which matters for support tickets and moderation. The speed and accuracy figures are the makers’, not ours.
• Pick by language and license first. Four of the five are English-first, one is non-commercial, and the one with a Turkish number is the smallest.
• Check your own data. The documentation warns the probabilities are not guaranteed to be calibrated for your data, and the announcement says to test confidence cutoffs on your own examples.

Editorial review: October 3, 2026 (AI). Prepared from ggml-org’s Hugging Face article, pull request 29818 (description, merged October 2), the server documentation, the GGUF model cards and Supersonic Labs’ Julia-1 card and results file (the Turkish figure was read from the published results file), all read the same day. Speed and accuracy figures are the makers’; we did not run llama.cpp or any model. The cover is ggml-org’s announcement card and the Julia 1 image is from its model card, both reproduced with credit and a link to the source. “What to keep in mind” is ours.