PostHog's Jeeves adds reasoning to Jev-style decision models, and pays for it in latency

Jeeves is a 9-billion-parameter model that appeared on Hacker News on September 29 and adds an explicit reasoning step to the kind of typed decision model we have covered in Jev and Laya-MLX, CLM-8B and OpenDecider. The date is that of the Hacker News submission, not a claim about when the repository was created. The code sits under PostHog’s GitHub organization, and the project’s citation names Nicholas P. Waltz as author.

The trade is the one you would expect. Jeeves reports better accuracy on harder decisions, and it pays for that in time. All figures below come from the project’s README, and we have not reproduced any of them.

What problem does it target?

Jev-style models return calibrated probabilities for a bounded choice, but according to the README that comes with lower accuracy, so many pipelines fall back to a large reasoning model for hard cases. Jeeves tries to close that gap by letting the decision model reason first.

It speaks the same request format as Jev, through a Jev-compatible API, and the repository ships a Python SDK meant as a drop-in replacement for Jev’s own. Three question types are supported in one request:

Question type	What it returns	Example use
Choice (choice)	A probability for each candidate	Route a message to returns, shipping or billing
Yes/no (noul)	A probability of yes	Decide whether a case needs urgent human attention
Score (score)	An expected level on an ordered scale	Rate how frustrated a customer is

How it works

Jeeves starts from Qwen3.5-9B and adds a LoRA adapter (rank 16) and a small pointer head. The model writes a reasoning chain, then a marker tells it to decide. The pointer head scores each option by comparing the model’s state at that marker with its state at the end of each option, and the scores become probabilities through a softmax with a temperature fitted on a development set. Put simply, the model chooses among the options it was given rather than generating an answer.

The README also records two design findings from ablations. The prompt marks its structure with rare, largely unused Qwen tokens, and using plain words such as “State” instead made results worse. So did not repeating the questions after the reasoning block.

What the results say

The README compares Jeeves with Jev and with Kev, an earlier open Jev-style model. A selection:

Measure (higher is better unless noted)	Kev-9B	Jev	Jeeves
Test overall (out-of-domain and held-out)	0.822	0.857	0.889
JevBench, 231 public items	0.715*	0.866	0.935
JevBench hard, 111 public items	0.451*	0.730	0.865
JevBench calibration error (lower is better)	n/a	0.049	0.037
MMLU	0.738	0.900	0.793
MMLU-Pro (10-way)	0.515	0.840	0.739
Transfer overall (MMLU-Pro and buried state)	0.579	0.800	0.746

* Kev-8B results. No Kev-9B JevBench result is published.

Jeeves leads on the held-out test split and on JevBench, and its calibration error is lower. Jev is clearly ahead on the knowledge-heavy MMLU and MMLU-Pro. The README says this itself.

Three details limit how far to read these numbers. The Kev-9B and Jev columns are the numbers Kev publishes, not results the Jeeves author reran. Outside JevBench, the README says the comparisons use different items drawn from the same sources. And the JevBench figures cover the public easy, standard and hard tiers only; the sealed judge tier is not included. Thinking itself matters: without it, the same checkpoint scores 0.804 on the 2,962-item test split, against 0.840 with it.

The price of thinking

Reasoning makes a request slower, and the README is candid about the tail. On 325 development questions, on one H100:

Setting	Accuracy	Mean reasoning tokens	Median / p90 latency
Full thinking	0.825	1,138	3.3 s / 17.1 s
max_think 768 with nothink_threshold 0.9	0.806	344	2.0 s / 5.6 s
No thinking	0.775	0	about 0.3 s

Two options do the work. max_think truncates each reasoning chain, and nothink_threshold skips reasoning when the quick answer is already confident enough. In the README’s own example, three questions in one request, thinking in parallel and capped at 512 tokens each, took about 8.1 seconds.

To speed up the reasoning itself, the project adds a diffusion drafter, an idea adapted from Orthrus. For a single question it raises chain generation from 109 to 176 tokens per second (1.6 times) at block size 4. Running Jeeves needs Python 3.12 and a CUDA GPU, with a Hopper-class card for the FP8 kernel.

Training and reproducibility

Training has three stages. Supervised fine-tuning runs for two epochs on 19,126 questions from 12 public datasets plus synthetic policy data, with about half of the questions carrying a reasoning chain sampled from the base model. Reinforcement learning with CISPO then uses 9,992 questions with eight rollouts each, capped at 2,560 thinking tokens. The README says the run was stopped at step 402 of a 624-step schedule because later steps over-sharpened the head. A single temperature is then fitted for calibration.

The code, the training pipeline and the data preparation scripts are released under the MIT license, and the weights are on Hugging Face. Each public dataset keeps its own license. We have not seen an independent reproduction.

Limits

• Knowledge questions trail Jev, as the table shows.
• The tail is slow. Full reasoning reaches 17 seconds at p90 unless you cap it.
• The reasoning is hard to inspect. The training had no language-consistency reward, so the chains are not well interpretable.
• The comparisons are the author’s. Nothing here has been checked by us or by an outside group.
• The hardware assumption is a data-center GPU. The quoted latencies are for an H100. That is a different setting from the local Apple Silicon story in our Laya-MLX article, or from the small footprints listed for OpenDecider on an L40S. Those figures use different hardware and are not directly comparable.

Our assessment

Jeeves is most interesting for pipelines that today send hard cases to a large reasoning model. There, a smaller model that reasons only when it is unsure could cut cost and delay, and nothink_threshold is the setting to test first. It looks less suited to knowledge-heavy classification, where the README’s own numbers favor Jev, and to paths where a 17-second tail is not acceptable.

If you evaluate it, use your own labelled items, give every candidate the same ones, and measure calibration as well as accuracy, since routing on confidence is the whole point. For the wider picture of Jev-style models, our earlier notes on CLM-8B and Jev Code Reviewer cover the alternatives and one applied example.

The cover is our own conceptual graphic and contains no data.