条件优先

Laya Benchmark：准确率、延迟与 Calibration 追踪

每个数字都保留来源、checkpoint、微调状态、数据集、样本量、硬件和测试条件。不同条件的数据不会被强行合并成一个排行榜。

特别注意：0.766 是 laya-typed-decisions 在对应 benchmark 家族上的 fine-tuned 结果，不是基础 Laya 的通用 zero-shot 准确率。

Upstream

Laya 上游测试

10 条记录

Benchmark	数据集	Checkpoint	测试条件	指标	结果	来源
Typed decisions Headline accuracy belongs to the specialized fine-tuned checkpoint, not base Laya zero-shot.	typed-decisions	`laya-typed-decisions` Laya	Tuning: Fine-tuned 样本: 2,000 decisions / 400 cases 硬件: T4 benchmark suite	Accuracy	0.766	Laya BENCHMARKS.md 2026-09-23
Typed decisions Base checkpoint sits below the reported majority-class baseline on this benchmark.	typed-decisions	`laya` Laya	Tuning: Base / zero-shot 样本: 2,000 decisions / 400 cases 硬件: T4 benchmark suite	Accuracy	0.361	Laya BENCHMARKS.md 2026-09-23
Application workflow Held-out task family in the upstream application benchmark.	LLM guardrails / jailbreak	`laya-typed-decisions` Laya	Tuning: Fine-tuned checkpoint 样本: 400 cases 硬件: CPU application benchmark	Accuracy	0.762	Laya BENCHMARKS.md 2026-09-23
Application workflow Held-out moderation remains weak; upstream explicitly warns against treating demo examples as production evidence.	Moderation / toxicity	`laya / typed-decisions` Laya	Tuning: Mixed checkpoints 样本: 400 cases 硬件: CPU application benchmark	Accuracy	0.530	Laya BENCHMARKS.md 2026-09-23
Application workflow Best upstream checkpoint in the RAG passage relevance theme.	RAG passage relevance	`laya-multilingual` Laya	Tuning: Base checkpoint 样本: 400 cases 硬件: CPU application benchmark	Accuracy	0.657	Laya BENCHMARKS.md 2026-09-23
Application workflow Ten-way support queue classification; this task family was in the training mix.	Support triage / 10-way queue	`laya-multilingual` Laya	Tuning: Base checkpoint 样本: 400 cases 硬件: CPU application benchmark	Accuracy	0.522	Laya BENCHMARKS.md 2026-09-23
Latency In-process model benchmark; not comparable to remote API latency without accounting for network and serving overhead.	1 question / call	`laya-multilingual` Laya	Tuning: Base checkpoint 样本: Warm-model latency run 硬件: Tesla T4	p50 latency	32.8 ms	Laya BENCHMARKS.md 2026-09-23
Latency v0.3.20, fp32, no GPU. About 185 ms per extra question up to 10 questions (1,842 ms for 10); batching questions saves little on CPU. Cold load about 2.5 s.	1 question / call	`laya-multilingual` Laya	Tuning: Base checkpoint 样本: 10 timed calls after 2 warm-up 硬件: Server CPU · AMD EPYC 9R14, 4 cores	p50 latency	193 ms	Laya BENCHMARKS.md 2026-09-25
Latency v0.3.20, fp32, no GPU. About 600 ms per question up to 10 questions (6,244 ms for 10), roughly 3x slower than laya-multilingual on the same CPU. Cold load about 4.4 s.	1 question / call	`laya` Laya	Tuning: Base checkpoint 样本: 10 timed calls after 2 warm-up 硬件: Server CPU · AMD EPYC 9R14, 4 cores	p50 latency	580 ms	Laya BENCHMARKS.md 2026-09-25
Latency v0.3.20, fp32, no GPU. Same per-question cost as the English checkpoint (6,031 ms for 10 questions). Peak memory for the whole run with several checkpoints loaded was 9.3 GiB.	1 question / call	`laya-typed-decisions` Laya	Tuning: Fine-tuned checkpoint 样本: 10 timed calls after 2 warm-up 硬件: Server CPU · AMD EPYC 9R14, 4 cores	p50 latency	584 ms	Laya BENCHMARKS.md 2026-09-25

Independent

独立第三方测试

6 条记录

Benchmark	数据集	Checkpoint	测试条件	指标	结果	来源
sysone-bench v2 Byte-identical inputs against Jev 1.13.0; the corresponding Jev measurement in the same run is 0.888.	Triage	`convaiinnovations/laya` Laya	Tuning: Base checkpoint 样本: 160 states 硬件: Local M2 CPU	Accuracy	0.800	sysone-bench 2026-09-23
sysone-bench v2 Byte-identical comparison; Jev measured 0.967 in the same benchmark run.	Guardrails	`convaiinnovations/laya` Laya	Tuning: Base checkpoint 样本: 60 states 硬件: Local M2 CPU	Accuracy	0.883	sysone-bench 2026-09-23
sysone-bench v2 Byte-identical comparison; Jev measured 0.910 in the same benchmark run.	AG News / 4 labels	`convaiinnovations/laya` Laya	Tuning: Base checkpoint 样本: 100 states 硬件: Local M2 CPU	Accuracy	0.940	sysone-bench 2026-09-23
sysone-bench v2 Byte-identical comparison; Jev measured 0.867 in the same benchmark run.	MNLI / 3-way NLI	`convaiinnovations/laya` Laya	Tuning: Base checkpoint 样本: 60 states 硬件: Local M2 CPU	Accuracy	0.983	sysone-bench 2026-09-23
sysone-bench v3 Router improves the multilingual result from 0.360 for the English checkpoint; Jev measured 1.000 in this small suite.	Multilingual intent / 5 languages	`Router-selected` Laya Router	Tuning: Base checkpoints + routing 样本: 25 states 硬件: Local M2 CPU	Accuracy	0.840	sysone-bench 2026-09-23
sysone-bench v2 Local Laya call latency reported by the benchmark. Jev API calls in the same run were 925–1068 ms.	5-question calls	`convaiinnovations/laya` Laya	Tuning: Base checkpoint 样本: 751-state benchmark run 硬件: Local M2 CPU	Latency	180–660 ms	sysone-bench 2026-09-23

Our test

本站复现

8 条记录

Laya、Jev、Kev、Von、GLiNER 在同样的英文和 8 种语言样本上的准确率、置信度校准和延迟。查看 System One 基准测试

Benchmark	数据集	Checkpoint	测试条件	指标	结果	来源
System One benchmark ECE 0.015, median 18.5 ms. Jev 1.13 scored 0.963 on the same examples.	SST-2 validation (2 labels)	`convaiinnovations/laya` Laya	Tuning: Base checkpoint, default settings 样本: 300 examples 硬件: Apple M5 Pro, PyTorch MPS	Accuracy	0.920	system-one-benchmark 2026-09-26
System One benchmark ECE 0.049, median 25.1 ms. Jev 1.13 scored 0.927 on the same examples.	TREC coarse test (6 labels)	`convaiinnovations/laya` Laya	Tuning: Base checkpoint, default settings 样本: 300 examples 硬件: Apple M5 Pro, PyTorch MPS	Accuracy	0.773	system-one-benchmark 2026-09-26
System One benchmark ECE 0.506: badly over-confident with 77 options. Jev 1.13 scored 0.813 on the same examples.	Banking77 test (77 labels)	`convaiinnovations/laya` Laya	Tuning: Base checkpoint, default settings 样本: 300 examples 硬件: Apple M5 Pro, PyTorch MPS	Accuracy	0.363	system-one-benchmark 2026-09-26
System One benchmark ECE 0.101, median 10.9 ms.	SST-2 validation (2 labels)	`convaiinnovations/laya (multilingual)` Laya multilingual	Tuning: Base checkpoint, default settings 样本: 300 examples 硬件: Apple M5 Pro, PyTorch MPS	Accuracy	0.857	system-one-benchmark 2026-09-26
System One benchmark ECE 0.062, median 14.9 ms.	TREC coarse test (6 labels)	`convaiinnovations/laya (multilingual)` Laya multilingual	Tuning: Base checkpoint, default settings 样本: 300 examples 硬件: Apple M5 Pro, PyTorch MPS	Accuracy	0.860	system-one-benchmark 2026-09-26
System One benchmark ECE 0.439, median 24.5 ms.	Banking77 test (77 labels)	`convaiinnovations/laya (multilingual)` Laya multilingual	Tuning: Base checkpoint, default settings 样本: 300 examples 硬件: Apple M5 Pro, PyTorch MPS	Accuracy	0.367	system-one-benchmark 2026-09-26
System One benchmark Upstream-recommended larger option budget; default settings scored 0.363. Jev 1.13 scored 0.813.	Banking77 test (77 labels)	`convaiinnovations/laya` Laya	Tuning: Base checkpoint, head_max_len=512, max_len=1024 样本: 300 examples 硬件: Apple M5 Pro, PyTorch MPS	Accuracy	0.427	system-one-benchmark 2026-09-26
System One benchmark Average over Chinese, Japanese, Korean, Spanish, German, Arabic and Hindi; median 20 ms. Jev 1.13 averaged 0.872, Kev-0.8B 0.656.	MASSIVE scenario, 7 non-English languages (18 labels)	`convaiinnovations/laya (multilingual)` Laya multilingual	Tuning: Base checkpoint, default settings 样本: 1,050 examples (150 per language) 硬件: Apple M5 Pro, PyTorch MPS	Accuracy	0.544	system-one-benchmark 2026-09-26

怎么读这些数字

不要跨条件直接比较

本地热模型延迟、远程 API 延迟、不同硬件和不同 batch 不能直接用倍数相除。

Fine-tuned 要单独标记

专门微调 checkpoint 的结果不能当成基础模型 zero-shot 能力。

优先用真实业务数据

最终阈值、confidence gating 与 fallback 策略都应该在自己的真实数据上重新验证。

提交 Benchmark