System One Benchmark: Jev vs Kev vs Laya vs Von vs GLiNER

System One models answer typed questions about a piece of text. Instead of writing an answer, they return a probability for every option. This System One benchmark runs them on three English datasets and on the same utterances in eight languages, asks every model the same questions, and publishes the code and every raw prediction so you can check or repeat the results.

Models tested: TypeSafe Jev 1.13 (hosted), Kev-0.8B, Von 1.2, Laya (English and multilingual checkpoints, and its automatic language router) and GLiNER2.5-Decide (English and multilingual versions). Code and raw results: system-one-benchmark on GitHub.

System One benchmark results at a glance

Jev 1.13 is at or near the top everywhere, in every language, and its confidence scores are the best calibrated.
Kev-0.8B is the strongest open model. It matches Jev on TREC and Banking77, but its training data includes those datasets (see Training-data overlap below). It is also the best open model on non-English input.
Laya multilingual is the fastest model tested, at about 20 ms per request, and it is the most accurate of the small encoder models (Laya, Von, GLiNER) on non-English input. It trails Jev and Kev on accuracy.
Laya stays weak with many labels: about 36% on Banking77's 77 intents at default settings, and 43% with the larger option budget its documentation recommends.
Von 1.2 is close to the leaders on English sentiment and Banking77, but weak on TREC and on non-English input, which its model card does not claim to support.
GLiNER2.5-Decide trails on these general datasets. It is tuned for its own suite of operational decisions.

English results

300 examples per dataset. Brackets show 95% confidence intervals (Wilson).

Model	SST-2 (2 labels)	TREC (6 labels)	Banking77 (77 labels)
Jev 1.13 (hosted)	0.963 [0.936–0.979]	0.927 [0.891–0.951]	0.813 [0.765–0.853]
Kev-0.8B †	0.920 [0.884–0.946]	0.953 [0.923–0.972]	0.823 [0.776–0.862]
Von 1.2	0.940 [0.907–0.962]	0.663 [0.608–0.714]	0.793 [0.744–0.835]
Laya (English)	0.920 [0.884–0.946]	0.773 [0.723–0.817]	0.363 [0.311–0.419]
Laya (multilingual)	0.857 [0.812–0.892]	0.860 [0.816–0.895]	0.367 [0.314–0.423]
GLiNER2.5-Decide	0.857 [0.812–0.892]	0.417 [0.362–0.473]	0.567 [0.510–0.622]
Majority-class baseline	0.500	0.263	0.030

† Kev's model card lists Banking77, TREC and SST-5 (the same sentences as SST-2) in its training data. We scored held-out test and validation examples, but Kev has seen these label sets and this kind of text.

Where intervals overlap, the gap is not established at this sample size. On Banking77, Jev, Kev and Von are statistically tied.

Multilingual benchmark results

The same 150 utterances from the MASSIVE test set in eight languages, sorted into 18 scenarios (alarm, weather, music, takeaway and so on). MASSIVE is a parallel corpus, so every language asks the same questions and the scores are directly comparable. The question and the option descriptions stay in English. None of the models lists MASSIVE as training data.

Model	English	Chinese	Japanese	Korean	Spanish	German	Arabic	Hindi	Non-English average
Jev 1.13 (hosted)	0.91	0.87	0.91	0.86	0.87	0.91	0.81	0.89	0.872 [0.851–0.891]
Kev-0.8B	0.76	0.77	0.75	0.69	0.62	0.71	0.56	0.49	0.656 [0.627–0.684]
Laya (multilingual)	0.65	0.66	0.63	0.48	0.59	0.50	0.47	0.49	0.544 [0.514–0.574]
Laya Router (automatic)	0.69	0.66	0.63	0.48	0.54	0.46	0.44	0.49	0.528 [0.497–0.558]
GLiNER2.5-Decide	0.77	0.61	0.65	0.43	0.65	0.65	0.29	0.13	0.488 [0.457–0.518]
GLiNER2.5-multi-Decide	0.55	0.46	0.50	0.46	0.45	0.56	0.40	0.09	0.416 [0.387–0.446]
Laya (English)	0.69	0.54	0.41	0.15	0.39	0.36	0.16	0.13	0.307 [0.280–0.335]
Von 1.2	0.73	0.15	0.13	0.09	0.31	0.25	0.13	0.11	0.168 [0.146–0.191]

Random guessing scores 0.056 on 18 scenarios. Mean calibration error (ECE) on the seven non-English languages: Jev 0.061, Kev 0.133, GLiNER2.5-Decide 0.141, GLiNER2.5-multi-Decide 0.195, Laya multilingual 0.207, Laya English 0.257 and Von 0.803.

What stands out:

Jev barely drops outside English. Every language stays between 0.81 and 0.91.
Laya multilingual keeps working across scripts. It is ahead of both GLiNER versions and Von on the non-English average, and it is the only small encoder that stays near 0.5 in Arabic and Hindi. The English checkpoint collapses on Korean, Arabic and Hindi, as the Laya project itself warns.
Force the multilingual checkpoint for non-English text. Laya's automatic router matched the multilingual checkpoint on Chinese, Japanese, Korean and Hindi, but scored a few points lower on Spanish, German and Arabic, likely because some of those requests were routed to the English checkpoint.
Von is English-only, as its model card says, and it stays confident when wrong outside English.

Laya's large-label settings on Banking77

Laya's documentation says 77 options share a fixed token budget in its decision head, and suggests two fixes. We ran both on the same 300 Banking77 examples:

Laya setting	English checkpoint	Multilingual checkpoint
Default	0.363	0.367
Larger option budget (`head_max_len=512`, `max_len=1024`)	0.427	0.417
Top-20 embedding shortlist (`predict_shortlist`)	0.280	0.300

The larger budget adds about 5 to 6 points and matches the 0.425 the Laya project reports. The shortlist, using the encoder's own mean-pooled embeddings, made things worse here: the correct intent often did not make the top 20. A stronger embedding model may work better. Either way, Laya remains far behind Jev (0.813) on this task without fine-tuning.

Calibration (ECE)

Expected calibration error (ECE) measures whether a model's confidence matches its accuracy. If a model says 90%, it should be right about 90% of the time. Lower is better. We use the probability of the top answer and 10 equal-width bins.

Model	SST-2	TREC	Banking77
Jev 1.13	0.020	0.020	0.083
Kev-0.8B †	0.032	0.030	0.128
Von 1.2	0.075	0.210	0.170
Laya (English)	0.015	0.049	0.506
Laya (multilingual)	0.101	0.062	0.439
GLiNER2.5-Decide	0.059	0.146	0.246

Calibration matters if you automate decisions above a confidence threshold. Jev's probabilities stay trustworthy even with 77 options and in other languages. Laya English is the best calibrated model on SST-2, but with many labels its stated confidence is far above its accuracy, so its raw probabilities should not drive automation there. The Laya project says the shipped checkpoints are over-confident and recommends fitting temperatures on your own data.

Latency

Median time per request, one request at a time, after five warm-up calls. Local models ran on an Apple M5 Pro Mac.

Model	Runs on	SST-2	Banking77	MASSIVE (Chinese)
Laya (multilingual)	Local, PyTorch MPS	10.9 ms	24.5 ms	20.1 ms
Laya (English)	Local, PyTorch MPS	18.5 ms	54.2 ms	36.6 ms
Von 1.2	Local, PyTorch MPS	17.6 ms	52.0 ms	31.8 ms
Kev-0.8B	Local, MLX	22.6 ms	68.9 ms	28.1 ms
GLiNER2.5-multi-Decide	Local, PyTorch MPS	–	–	36.3 ms
GLiNER2.5-Decide	Local, PyTorch MPS	26.7 ms	1,203 ms	86.3 ms
Jev 1.13	Hosted, through OpenRouter	410 ms	409 ms	375 ms

Jev's figure is end-to-end over the internet, network round trip included. It tells you what a remote call costs in practice, not how fast the model itself is. For the local models, time grows with the number of options because every option is encoded.

Where Laya fits

These results put Laya in a clear niche rather than at the top of the table:

Latency-critical, high-volume decisions with a handful of options, where 10 to 20 ms locally matters more than the last few points of accuracy.
Many languages on small hardware. Among the small encoders we tested, Laya multilingual is the one that works across scripts.
Fine-tuning on your own data. The Laya project reports that its fine-tuned checkpoint jumps from 0.362 to 0.766 on its typed-decisions benchmark. Zero-shot scores like the ones above are a starting point. See Fine-tune Laya.
Not for large label sets out of the box. With 50+ options, use Jev or Kev, or split the question into coarse and fine steps.

Method

English datasets. SST-2 validation split (movie-review sentiment), TREC coarse test split (question type) and the official Banking77 test split from PolyAI (customer-support intent). 300 examples each, sampled once with seed 42.
Multilingual dataset. MASSIVE scenario classification, test split: 150 utterances sampled once by ID from English, then the same IDs in Chinese (zh-CN), Japanese, Korean, Spanish, German, Arabic and Hindi.
Questions. Each dataset becomes one choice question with the same English instruction and option descriptions for every model and language.
Interface. Every model is called with the same Jev-style POST /v1/systemone request: Laya through laya-serve (with a small wrapper for the two large-label settings), Von through von serve, Kev through its own server, and Jev through OpenRouter's System One endpoint. GLiNER2.5 has no such API, so a small shim maps the request onto its classify_text call.
No tuning by us. No fine-tuning, prompt tuning or recalibration. Each model runs with its release defaults, except the Laya settings named above.
Versions. Jev typesafe/jev-1.13 on OpenRouter; Kev jaredpalmer/kev-0.8b (MLX, bfloat16); Von 1.2 (von-sdk 1.2.3); Laya 0.3.20; GLiNER2.5-Decide and GLiNER2.5-multi-Decide with gliner2 2.0.0. Run on September 26, 2026.

Training-data overlap

A public dataset in a model's training mix inflates its score on that dataset. We checked each model's published documentation:

Model	Trained on these datasets?
Kev-0.8B	Yes for Banking77, TREC and SST-5. MASSIVE is not listed.
Laya	Not listed. Upstream lists AG News and BoolQ as training data and SST-5 as held out. It reports its own MASSIVE intent results.
Von 1.2	The model card does not list its training data.
GLiNER2.5-Decide	Not listed. It is evaluated on its own fast-decisions suite.
Jev 1.13	Not disclosed.

"Not listed" is not proof that a dataset was unused.

Limitations

Small samples. 300 examples per English dataset and 150 per language: gaps of a few points sit inside the error bars.
One run, one machine. Latency will differ on other hardware.
choice questions only. Yes/no (noul) and ordinal (score) questions are not covered.
One wording, in English. Other instructions, option descriptions or questions written in the input's language can move the results.
Zero-shot only. Fine-tuning changes the picture, especially for Laya, and is not measured here.
Smallest Kev only. Kev also ships 4B, 9B and 27B checkpoints.

Reproduce the System One benchmark

git clone https://github.com/yanng981/system-one-benchmark
cd system-one-benchmark
python prepare_data.py                               # writes the fixed samples
python bench.py laya von-1.2 kev-0.8b                # English datasets
python bench.py laya-multilingual kev-0.8b --datasets multilingual
OPENROUTER_API_KEY=... python bench.py jev --datasets multilingual

The repository README explains how to start each model server. Raw predictions for every model and dataset are in results/.

FAQ

Which System One model is most accurate?

In this benchmark, Jev 1.13: it has the most consistent accuracy and the best calibration in every task and language. Kev-0.8B is the strongest open model.

Which System One model is best for multilingual input?

Jev, by a wide margin (0.87 average outside English). Among open models, Kev-0.8B leads (0.66), followed by Laya multilingual (0.54), which is also the fastest.

Is Laya as accurate as Jev?

No. On two-label sentiment Laya English is within a few points of Jev and answers locally in about 20 ms, but with many labels or non-English input Jev is far ahead. Laya's advantages are speed, open weights, running on your own hardware and fine-tuning. See Laya vs Jev.

Does Laya's larger option budget fix Banking77?

Partly. head_max_len=512 lifts Laya from about 0.36 to about 0.43 on Banking77's 77 intents. That is still far below Jev's 0.81.

What is Jev?: how Jev works, pricing and API access
System One models compared: every model side by side
Kev and Von: model guides
Laya vs Jev: open weights, speed, accuracy and tradeoffs
Laya benchmark tracker: upstream and independent Laya results

Measured September 26, 2026.