Jeff shrinks Jev-style decision models to 0.8B and answers in about 22 ms

Jeff is a set of three small open models that appeared on Hacker News on September 28 and answer in the request format of Jev, the typed decision model we have covered in Jev and Laya-MLX. Where PostHog’s Jeeves adds reasoning and pays in latency, Jeff goes the other way: it shrinks the model and answers in tens of milliseconds. The project says it is independent and not affiliated with or endorsed by TypeSafe, the makers of Jev. The figures below come from the project’s README, and we have not reproduced any of them.

What it is

Jeff is three fine-tunes: Jeff-Qwen3.5-0.8B, Jeff-Qwen3.5-2B and Jeff-Gemma4-E2B. You describe a situation and list options in plain words, and Jeff returns a calibrated probability for each option from a single forward pass, with no generated text to parse. It supports the same three question types as Jev: choice, yes/no (noul) and score.

It began as a fork of AutoJev, an open MIT-licensed recipe that fine-tunes Qwen3.8-27B to give Jev-style decisions, and shrinks that idea to much smaller students. Code is MIT and the weights are Apache 2.0. The training data is not released, and some of its sources are share-alike.

Speed and size

The README measures the median time per decision over the same 200 benchmark questions, about 200 input tokens each, one at a time, from raw text to probabilities.

Model	Weights (16-bit)	RTX PRO 6000	Apple M4 Max (MLX)	CPU (32 threads)
Jeff-Qwen3.5-0.8B	1.7 GB	22 ms	28 ms	463 ms
Jeff-Qwen3.5-2B	4.2 GB	24 ms	60 ms	708 ms
Jeff-Gemma4-E2B	9.3 GB	29 ms	not available	1.0 s

For comparison, the README notes that Jev’s published Doom runs took 114 to 212 ms per call including the network. The two were not measured on the same hardware, so treat that as context, not a ranking.

Accuracy against Jev

The README tests Jeff on 4,599 questions from five public benchmarks, plus the public hard tier of JevBench, and sets the results beside Jev’s published figures. The Qwen columns are v1.1; the Gemma model is still v1.0.

Benchmark	Jeff 0.8B	Jeff 2B	Jeff Gemma	Jev (published)
Overall (five benchmarks)	79.1	82.0	81.6	83.0
Financial PhraseBank	95.7	94.7	96.1	77.0
RAGTruth	85.6	87.7	87.4	77.3
BBH	64.9	68.7	66.4	94.3
JudgeBench	63.1	59.4	60.6	78.6
WinoGrande	69.0	78.8	77.4	90.7
JevBench hard (105 items)	46.7	57.1	48.6	73.3

The pattern is what you would expect at this size. Jeff matches or beats Jev on classification and grounding tasks, and it stays well below it on reasoning-heavy ones. The README says so itself. The caveat that matters: Jev’s numbers were measured on a different sample of the same benchmarks, so the gaps are indicative, not exact. Note also that this README counts 105 items in the public hard tier, while Jeeves’ README counts 111, another reason not to lay numbers from different projects side by side.

The v1.1 fix

The most interesting detail is a bug. v1.0, released on September 28, never picked an option past the 26th, because no training question had more than 19 options. A user reported it, and on September 29 the project shipped v1.1 with 32,000 new training questions of 20 to 254 options. On the long-list test, the 0.8B model went from 40.3% to 94.7%, and the 2B to 95.2%. Calibration error for the 0.8B fell from 0.049 to 0.021.

It came with trade-offs. The 2B’s overall score dipped from 83.1% to 82.0%, so it no longer edges past Jev’s published 83.0%, and two datasets, MASSIVE and CLINC150, are now in the training data, so they are no longer zero-shot tests. The Gemma model was not retrained and still handles at most 26 options. The chart in the repository still shows the older v1.0 figures, so use the README’s table.

Built on local hardware

The author says training used one RTX PRO 6000: about two hours for the 0.8B model and three and a half for the 2B. The synthetic training data was written by an open model (Qwen3.8-Flash-Next) on two DGX Sparks, and testing ran on a MacBook. No cloud GPUs were used, and no closed-model output is in the training data; a closed model only spot-checked the quality of a sample.

Beyond classification: games and chess

To probe zero-shot behavior, the README has Jeff play three games (measured with v1.0), describing the state and the legal moves in words each turn. The 0.8B model scored 6.55 kills in Doom, 10.3 crossings in Frogger and 57.0 of 98 pellets in Pac-Man, roughly matching a hand-coded rule bot on the first two. It decided in 29 to 49 ms per move on an M4 Max.

A chess fine-tune shows the other use of speed. Trained for about three and a half hours on 600,000 positions labelled by Stockfish, the 0.8B model solved 55.8% of 1,000 held-out puzzles, up from 15.5% zero-shot. It is a weak player, about 1,000 Elo with no search, and the clip below is its one win out of 100 simultaneous games against Stockfish’s weakest setting.

Limits

• Small models do not reason. Expect fast, calibrated choices between options you describe, not multi-step reasoning. The README says this holds for every model of this size.
• Wording matters a lot. In Frogger, giving the goal option the same words as the others took one episode from 15 crossings to 23.
• Benchmark scores do not predict behavior. The untrained Gemma model beats the Qwen models on benchmarks yet plays the games worst. The 2B scores higher than the 0.8B on benchmarks but plays worse.
• English and text only, and the comparisons with Jev are the author’s, on different samples.

Our assessment

Jeff looks most useful where latency and cost dominate and the categories are plain: routing a message to a queue, tagging intent, moderation labels, on-device or offline use. For those, a 1.7 GB model that answers in about 22 ms is a different proposition from a reasoning model that answers in seconds. It is a weaker fit for tasks that need real reasoning, where Jev, and models like Jeeves, are stronger.

If you evaluate it, test on your own labelled items, give every candidate the same ones, and check calibration as well as accuracy. Also test your option lists at the size you will really use, since long lists were exactly what v1.0 got wrong. For other open options in this family, see our notes on OpenDecider and Jeeves.

The cover is an AI-generated illustration and not an official design. The chess clip is the project’s own demo.