x.com
Morgan on X: It has been a really interesting experience to build an eval suite for System On
@morganlinton@morganlintonlisted 1h agoreviewed by JevIt has been a really interesting experience to build an eval suite for System One Models like Jev. My first two attempts didn't work out. But I learned a lot in the process, and since Jev is so fast, and cheap, it's a lot easier to run quick tests than with an LLM. I'm on attempt number three with this eval suite, and thought I'd share more about it. The eval suite is called @VulcanBench Verdict, and here's the high-level on how it works: ❤️ 32 likes on X
- Author
- @morganlinton
- Use case
- Benchmarks & Evals
- Added
- 2026-09-25
All figures come from the author. Check the source before you quote them.
More in Benchmarks & Evals
- silentguy on X: Grok Bot does the job, Jev decides where the job goes next▲ 0x.com
- spect on X: The founder of Jev just dropped a 1-hour masterclass on how Jev actually works▲ 0x.com
- Dain on X: A beautiful pattern can still be noise.▲ 0x.com
- Utkarsh Maheshwari on X: Is the Jev hype real,▲ 0x.com
- OpenMed on X: Four typed questions across four authored fictional notes, labels written before▲ 0x.com
- JevBench v1.4.1 — frozen Jev alternatives ranking▲ 0benchmarkheaven.com
Jev guides for this use case
- Using Jev as a judge for evalsGrading with a decision model instead of a prose-writing judge, and why calibrated confidence is the real prize.
- How Jev sorts a build into one of 21 use casesThe 21 criteria Jev classifies against, published in full, plus what the reviewer sees and how ambiguity is handled.
- Jev statistics: latency, cost, and this directory's own numbersPublished benchmarks with their caveat attached, plus live directory figures that update automatically.
Bid history
No bids yet — the first one takes this project straight to the spotlight.