Using Jev as a judge for evals

“Jev as a judge” means using it to grade outputs — a response, a diff, a candidate — rather than to produce them. It is the same shape as LLM-as-a-judge, minus the part where you parse prose and hope the model followed your rubric.

Updated September 2026 · 6 min read

The standard problem with LLM-as-a-judge

Using a large model to grade outputs is well-established, and its known weaknesses are all downstream of one fact: the judge writes prose.

You have to parse it. A rubric in, a paragraph out. Getting a number means regexes, JSON instructions, or a second model call.
Format compliance becomes a failure mode. Some fraction of your judgments fail not because the judge disagreed but because it explained itself instead of answering.
Self-preference. Judges tend to favour text that resembles their own. A judge that does not write has no such bias.
No usable uncertainty. Prose confidence (“this is likely good”) is not something you can threshold on.

Why a decision model fits grading

Grading is already a closed-set question in most rubrics. You have dimensions, and each one has a scale. That is score for the ordered dimensions and choice for the categorical ones — the two primitives that exist for exactly this.

Rubric step	Primitive	What you get back
Is this output acceptable?	`noul`	A probability you can threshold and record
Rate this along each dimension	`score`	A fractional score per dimension, plus the distribution
Which failure mode is this?	`choice`	One of your defined failure categories

Because the answer set is declared, a judge run cannot produce an unparseable verdict. Every judgment is a number or a key, ready to aggregate.

Calibration is the actual prize

Jev's training method is RLCD — reinforcement learning for calibrated decisions. The claim is not that it is right more often; it is that when it says 0.9, it means roughly 0.9.

For evals that matters more than raw accuracy. A judge whose confidences mean something lets you:

Sample intelligently. Send only the uncertain cases to human review, instead of a flat percentage of everything.
Detect drift. If the distribution of confidences shifts between model versions, something changed — even when the mean score did not.
Set thresholds you can defend. “Auto-pass above 0.95” is a policy. “The model seemed confident” is not.

Calibration is a claim, not a guarantee

Calibration is measured on a distribution, and it degrades on inputs unlike the training data. Before trusting confidences for auto-approval, measure them on your own held-out set — the same discipline you would apply to any judge.

We already use it this way

This site's submission review is a Jev-as-a-judge pipeline running in production. One call asks whether the link is a Jev build at all and which of the 21 use cases it belongs to.

The verdict is a value, not a text. No parsing layer, no format-compliance failures to handle.
The downgrade is visible. When the reviewer is unavailable (no API key, network failure), submissions fall back to a reachability heuristic and are labeled reviewed by fallback instead of reviewed by Jev. Each listing carries a badge saying which reviewer ran.
It is cheap enough to run on everything. We judge every submission rather than sampling, which is only affordable because a judgment costs a fraction of a cent.

The badge system is the part worth copying if you build something similar: the criteria and the pipeline are documented here.

Evaluation tooling in this directory

Listings Jev sorted into Benchmarks & Evals:

Accuracy figures quoted above are TypeSafe's published numbers. Reference answers in these evals came from two large models, not humans.

Common questions

What does 'Jev as a judge' mean?

Using Jev to grade outputs rather than produce them. You supply the rubric as typed questions — a score per dimension, or a choice among failure modes — and get numbers back instead of prose.

How is this better than LLM-as-a-judge?

No parsing step and no format-compliance failures, because the answer set is declared up front. A judge that does not write text also has no self-preference bias toward fluent answers.

What does 'calibrated' mean here?

That a stated 0.9 probability corresponds to roughly a 90% chance of being correct. Calibration is what makes confidence thresholds usable for auto-approval and selective human review.

Should I trust Jev's confidence for auto-approval?

Measure it on your own held-out set first. Calibration is a distribution-level property and degrades on inputs unlike the training data, so the same discipline applies as with any judge.

BuiltOnJev is an independent community project, not affiliated with TypeSafe AI. Specs and eval numbers come from TypeSafe's published materials; directory figures come from this site's own submissions and are updated as builds arrive.