Red Hat: Jev is competitive with guardrail baselines, not clearly better

The decision-model idea behind Jev, Laya and similar models promises fast, cheap, schema-safe answers without generating text. Red Hat’s AI Safety team has now put three of them against the tools they are meant to replace. In an October 2 article (page marked updated October 5), Rob Geada, Mac Misiura and Shelton Cyril report that decision models were competitive, but not reliably better. This is an independent test of a model family we have covered before, with Red Hat’s own tilt noted below. We read the article and its appendix; we did not rerun anything.

What did Red Hat test?

Red Hat built guardrails that decide whether a prompt should be blocked, using two benchmarks from EvalHub’s NeMo Guardrails library: prompt injection and content safety (toxicity and profanity). Both datasets are balanced between risky and safe prompts, and all prompts are in English.

• Decision-style: Jev 1.13.0 through TypeSafe’s API, Laya (ModernBERT-based, open source) and DiffusionGemma 26B on an experimental /v1/systemone endpoint in vLLM.
• LLM-as-a-judge: Qwen3.6-35B, NVIDIA’s Nemotron-3.5-Content-Safety (with default and custom policies) and Mistral’s Shieldstral-1.0.
• Pre-trained classifiers: Red Hat’s deberta-v3-base for prompt injection and granite-guardian-hap-125m for content safety, the models in OpenShift AI 3.6’s default guardrail catalog.
• Baseline: BART-large-mnli, a 2019 zero-shot classifier.

The classifiers, Laya and BART ran on a MacBook Pro M1 CPU. The other models ran on a 96GB GPU node in a US cloud region, and Jev through its remote API, all measured from the UK.

How did Jev, Laya and DiffusionGemma score?

Method	Prompt injection	Content safety	Median latency, injection
Qwen3.6-35B (judge)	89.31%, 1st	85.47%, 3rd	312.5 ms
deberta-v3-base (classifier)	89.01%, 2nd	not used	54.1 ms
DiffusionGemma	87.72%, 3rd	85.53%, 2nd	561.7 ms
Jev	86.35%, 4th	86.20%, 1st	348.1 ms
Laya	85.44%, 5th	57.87%, 9th	119.3 ms
Nemotron-3.5, custom policy	84.84%, 6th	85.07%, 4th	240.4 ms
granite-guardian-hap-125m (classifier)	not used	80.27%, 6th	33.2 ms (safety run)
BART-large-mnli	61.49%, 9th	68.67%, 8th	115.0 ms

Shieldstral and the default-policy Nemotron are left out for space; see the article. Rankings and figures are Red Hat’s.

• Classifiers hold up. On prompt injection the 200M-parameter classifier finished 0.20 points behind first place at about a sixth of the leader’s latency. On content safety, Jev and DiffusionGemma beat Red Hat’s small granite classifier clearly, which the authors read as a sign that their small safety classifiers need improving.
• Jev’s result is split. It led content safety by 0.67 points over DiffusionGemma, and was fourth on prompt injection, 2.96 points behind the top.
• The open alternatives compete. DiffusionGemma finished within about 1.4 points of Jev on both tasks. Laya was within four points of the top on prompt injection and far behind on content safety.
• Prompts matter per model. With a policy Red Hat tuned for it, Laya rose from 57.87% to 75.20%, while Jev dropped from 86.20% to 82.53% on that same policy. The authors conclude that a prompt style that suits one model does not necessarily suit another.
• One judge underperformed. Red Hat says Shieldstral did much worse than Mistral’s own benchmarks suggest and needed several rounds of prompt work; with the shared risk definitions it lost about 10 points.

Red Hat’s conclusion: pre-trained classifiers “remain extremely competitive,” and decision models “do not reliably outperform” LLM judges, pre-trained classifiers or open decision models in speed or accuracy. It credits Laya’s compact size as a real advantage if you can prompt around its limits, and welcomes the shift toward small, task-specific models.

What are the caveats?

• Red Hat has a position. The authors say the AI Safety team has “always advocated” small predictive models, and the article notes that these are what OpenShift AI 3.6 ships by default. That does not make the numbers wrong, but the conclusion matches the vendor’s product.
• Latency is not like for like. The classifiers and Laya ran locally on a laptop CPU; Jev and the GPU-hosted judges were remote calls with about 56 ms of transatlantic delay at minimum, as Red Hat itself notes. The classifier’s 54 ms against Jev’s 348 ms mixes model speed with network distance. Jev’s own accuracy figures are not affected.
• The risk definitions came from LLM judges. Red Hat adapted them from prompts it uses with LLM-as-a-judge guardrails and says they may not be optimal for decision models. That could understate Jev, Laya and DiffusionGemma.
• Narrow scope. Two tasks, English only, the article does not say how many runs were made and reports no confidence intervals, and Jev’s parameter count is listed as unknown. It did not test cost or other languages, and Red Hat says it would like to test OpenAI’s Decisions API, which is in limited preview.
• Its tables disagree in places. Shieldstral’s accuracy appears as 72.02% and 74.80% in the two main tables and the reverse in the appendix, so we cannot say which is right, and the chart on this page marks it. The article also calls its field “nine candidate guardrails” in one place and “eight” in another, and lists DiffusionGemma as about 2,600 million parameters although its name says 26B.

What to keep in mind

These are our observations:

• It does not refute the idea. A decision model that matches an LLM judge at a lower cost would still be useful, but this test did not measure cost and found no speed advantage over the judges it hosted remotely.
• It fits other independent signals. Our earlier roundup and the llama.cpp release both stressed testing a model on your own data before trusting its probabilities. This benchmark supports that advice.
• Test your own task. The ranking flipped between the two tasks here, so the best choice depends on the guardrail you are building.

Editorial review: October 5, 2026 (AI). Prepared from Red Hat Developer’s article of October 2, 2026 (page marked last updated October 5), including its appendix tables and prompts, read in full the same day through a browser, since a plain request was blocked. We did not rerun the benchmarks or test the models. Figures are Red Hat’s; Shieldstral values are shown as in the main tables and the discrepancy is noted. The cover is our own chart built from Red Hat’s published accuracy figures, credited to Red Hat. “Read it with these caveats” and “What to keep in mind” are ours.