Context first

Laya Benchmarks: Accuracy, Latency & Calibration Tracker

Every number keeps its source, checkpoint, tuning status, dataset, sample size, hardware, and test conditions. Results from different setups are not forced into one leaderboard.

Important: 0.766 belongs to the fine-tuned laya-typed-decisions checkpoint on its benchmark family. It is not generic base-Laya zero-shot accuracy.

Submit a benchmark

Upstream

Laya upstream measurements

10 records

Benchmark	Dataset	Checkpoint	Conditions	Metric	Value	Source
Typed decisions Headline accuracy belongs to the specialized fine-tuned checkpoint, not base Laya zero-shot.	typed-decisions	`laya-typed-decisions` Laya	Tuning: Fine-tuned Sample: 2,000 decisions / 400 cases Hardware: T4 benchmark suite	Accuracy	0.766	Laya BENCHMARKS.md 2026-09-23
Typed decisions Base checkpoint sits below the reported majority-class baseline on this benchmark.	typed-decisions	`laya` Laya	Tuning: Base / zero-shot Sample: 2,000 decisions / 400 cases Hardware: T4 benchmark suite	Accuracy	0.361	Laya BENCHMARKS.md 2026-09-23
Application workflow Held-out task family in the upstream application benchmark.	LLM guardrails / jailbreak	`laya-typed-decisions` Laya	Tuning: Fine-tuned checkpoint Sample: 400 cases Hardware: CPU application benchmark	Accuracy	0.762	Laya BENCHMARKS.md 2026-09-23
Application workflow Held-out moderation remains weak; upstream explicitly warns against treating demo examples as production evidence.	Moderation / toxicity	`laya / typed-decisions` Laya	Tuning: Mixed checkpoints Sample: 400 cases Hardware: CPU application benchmark	Accuracy	0.530	Laya BENCHMARKS.md 2026-09-23
Application workflow Best upstream checkpoint in the RAG passage relevance theme.	RAG passage relevance	`laya-multilingual` Laya	Tuning: Base checkpoint Sample: 400 cases Hardware: CPU application benchmark	Accuracy	0.657	Laya BENCHMARKS.md 2026-09-23
Application workflow Ten-way support queue classification; this task family was in the training mix.	Support triage / 10-way queue	`laya-multilingual` Laya	Tuning: Base checkpoint Sample: 400 cases Hardware: CPU application benchmark	Accuracy	0.522	Laya BENCHMARKS.md 2026-09-23
Latency In-process model benchmark; not comparable to remote API latency without accounting for network and serving overhead.	1 question / call	`laya-multilingual` Laya	Tuning: Base checkpoint Sample: Warm-model latency run Hardware: Tesla T4	p50 latency	32.8 ms	Laya BENCHMARKS.md 2026-09-23
Latency v0.3.20, fp32, no GPU. About 185 ms per extra question up to 10 questions (1,842 ms for 10); batching questions saves little on CPU. Cold load about 2.5 s.	1 question / call	`laya-multilingual` Laya	Tuning: Base checkpoint Sample: 10 timed calls after 2 warm-up Hardware: Server CPU · AMD EPYC 9R14, 4 cores	p50 latency	193 ms	Laya BENCHMARKS.md 2026-09-25
Latency v0.3.20, fp32, no GPU. About 600 ms per question up to 10 questions (6,244 ms for 10), roughly 3x slower than laya-multilingual on the same CPU. Cold load about 4.4 s.	1 question / call	`laya` Laya	Tuning: Base checkpoint Sample: 10 timed calls after 2 warm-up Hardware: Server CPU · AMD EPYC 9R14, 4 cores	p50 latency	580 ms	Laya BENCHMARKS.md 2026-09-25
Latency v0.3.20, fp32, no GPU. Same per-question cost as the English checkpoint (6,031 ms for 10 questions). Peak memory for the whole run with several checkpoints loaded was 9.3 GiB.	1 question / call	`laya-typed-decisions` Laya	Tuning: Fine-tuned checkpoint Sample: 10 timed calls after 2 warm-up Hardware: Server CPU · AMD EPYC 9R14, 4 cores	p50 latency	584 ms	Laya BENCHMARKS.md 2026-09-25

Independent

Independent measurements

6 records

Benchmark	Dataset	Checkpoint	Conditions	Metric	Value	Source
sysone-bench v2 Byte-identical inputs against Jev 1.13.0; the corresponding Jev measurement in the same run is 0.888.	Triage	`convaiinnovations/laya` Laya	Tuning: Base checkpoint Sample: 160 states Hardware: Local M2 CPU	Accuracy	0.800	sysone-bench 2026-09-23
sysone-bench v2 Byte-identical comparison; Jev measured 0.967 in the same benchmark run.	Guardrails	`convaiinnovations/laya` Laya	Tuning: Base checkpoint Sample: 60 states Hardware: Local M2 CPU	Accuracy	0.883	sysone-bench 2026-09-23
sysone-bench v2 Byte-identical comparison; Jev measured 0.910 in the same benchmark run.	AG News / 4 labels	`convaiinnovations/laya` Laya	Tuning: Base checkpoint Sample: 100 states Hardware: Local M2 CPU	Accuracy	0.940	sysone-bench 2026-09-23
sysone-bench v2 Byte-identical comparison; Jev measured 0.867 in the same benchmark run.	MNLI / 3-way NLI	`convaiinnovations/laya` Laya	Tuning: Base checkpoint Sample: 60 states Hardware: Local M2 CPU	Accuracy	0.983	sysone-bench 2026-09-23
sysone-bench v3 Router improves the multilingual result from 0.360 for the English checkpoint; Jev measured 1.000 in this small suite.	Multilingual intent / 5 languages	`Router-selected` Laya Router	Tuning: Base checkpoints + routing Sample: 25 states Hardware: Local M2 CPU	Accuracy	0.840	sysone-bench 2026-09-23
sysone-bench v2 Local Laya call latency reported by the benchmark. Jev API calls in the same run were 925–1068 ms.	5-question calls	`convaiinnovations/laya` Laya	Tuning: Base checkpoint Sample: 751-state benchmark run Hardware: Local M2 CPU	Latency	180–660 ms	sysone-bench 2026-09-23

Our test

Our reproducible tests

8 records

Laya, Jev, Kev, Von and GLiNER on the same examples in English and eight other languages, with accuracy, calibration and latency. Read the System One benchmark

Benchmark	Dataset	Checkpoint	Conditions	Metric	Value	Source
System One benchmark ECE 0.015, median 18.5 ms. Jev 1.13 scored 0.963 on the same examples.	SST-2 validation (2 labels)	`convaiinnovations/laya` Laya	Tuning: Base checkpoint, default settings Sample: 300 examples Hardware: Apple M5 Pro, PyTorch MPS	Accuracy	0.920	system-one-benchmark 2026-09-26
System One benchmark ECE 0.049, median 25.1 ms. Jev 1.13 scored 0.927 on the same examples.	TREC coarse test (6 labels)	`convaiinnovations/laya` Laya	Tuning: Base checkpoint, default settings Sample: 300 examples Hardware: Apple M5 Pro, PyTorch MPS	Accuracy	0.773	system-one-benchmark 2026-09-26
System One benchmark ECE 0.506: badly over-confident with 77 options. Jev 1.13 scored 0.813 on the same examples.	Banking77 test (77 labels)	`convaiinnovations/laya` Laya	Tuning: Base checkpoint, default settings Sample: 300 examples Hardware: Apple M5 Pro, PyTorch MPS	Accuracy	0.363	system-one-benchmark 2026-09-26
System One benchmark ECE 0.101, median 10.9 ms.	SST-2 validation (2 labels)	`convaiinnovations/laya (multilingual)` Laya multilingual	Tuning: Base checkpoint, default settings Sample: 300 examples Hardware: Apple M5 Pro, PyTorch MPS	Accuracy	0.857	system-one-benchmark 2026-09-26
System One benchmark ECE 0.062, median 14.9 ms.	TREC coarse test (6 labels)	`convaiinnovations/laya (multilingual)` Laya multilingual	Tuning: Base checkpoint, default settings Sample: 300 examples Hardware: Apple M5 Pro, PyTorch MPS	Accuracy	0.860	system-one-benchmark 2026-09-26
System One benchmark ECE 0.439, median 24.5 ms.	Banking77 test (77 labels)	`convaiinnovations/laya (multilingual)` Laya multilingual	Tuning: Base checkpoint, default settings Sample: 300 examples Hardware: Apple M5 Pro, PyTorch MPS	Accuracy	0.367	system-one-benchmark 2026-09-26
System One benchmark Upstream-recommended larger option budget; default settings scored 0.363. Jev 1.13 scored 0.813.	Banking77 test (77 labels)	`convaiinnovations/laya` Laya	Tuning: Base checkpoint, head_max_len=512, max_len=1024 Sample: 300 examples Hardware: Apple M5 Pro, PyTorch MPS	Accuracy	0.427	system-one-benchmark 2026-09-26
System One benchmark Average over Chinese, Japanese, Korean, Spanish, German, Arabic and Hindi; median 20 ms. Jev 1.13 averaged 0.872, Kev-0.8B 0.656.	MASSIVE scenario, 7 non-English languages (18 labels)	`convaiinnovations/laya (multilingual)` Laya multilingual	Tuning: Base checkpoint, default settings Sample: 1,050 examples (150 per language) Hardware: Apple M5 Pro, PyTorch MPS	Accuracy	0.544	system-one-benchmark 2026-09-26

How to read the tracker

Do not compare unlike setups

Warm local inference, remote API latency, different hardware, and different batch sizes are not directly interchangeable.

Keep fine-tuning visible

A specialized fine-tuned checkpoint should not be presented as base-model zero-shot capability.

Validate on your workload

Thresholds, confidence gating, and fallback policy should be validated on representative application data.

Submit a benchmark