Laya on CPU: Latency, Threads & Sizing (4-Core Benchmarks)

Can you run Laya without a GPU? Yes, and the upstream project now publishes enough CPU measurements to say how fast, what it costs per question, and which settings matter. This guide reads those numbers so you can size a CPU deployment before you buy hardware.

All figures below come from the upstream Laya BENCHMARKS.md. Each is tied to the hardware and setup it was measured on.

The short answer

On a 4-core server CPU, laya-multilingual answers one question in about 193 ms; the English and typed-decisions checkpoints take about 580 ms.
On CPU, cost grows almost linearly with the number of questions. Batching questions into one call saves little, unlike on a GPU.
Thread settings can matter more than the model. On one laptop, fixing torch's thread defaults made calls about 12x faster.
Cold loads take seconds, and several loaded checkpoints can use around 9 GiB of RAM, so preload what you serve.

The server CPU numbers

Measured in-process on an AWS m7a.xlarge (AMD EPYC 9R14, 4 physical cores, no SMT, 16 GiB RAM), Laya v0.3.20, fp32, OMP_NUM_THREADS=4. Each call alternates a 3-option choice and a noul question. Values are p50; p95 was within 2% of p50 on every row.

checkpoint	1 question	5	10	50	cold load
`laya` (English)	580 ms	3,072 ms	6,244 ms	35,969 ms	4.4 s
`laya-multilingual`	193 ms	912 ms	1,842 ms	11,157 ms	2.5 s
`laya-typed-decisions`	584 ms	2,819 ms	6,031 ms	35,653 ms	0.5 s

Two things stand out.

The multilingual checkpoint is about 3x faster on CPU. It is built on mmBERT-base (322M parameters) rather than ModernBERT-large (421M). The upstream write-up does not break down where the rest of the gap comes from, so treat 3x as a measured result on this machine, not a rule.

The p95 is almost the same as the p50. On an otherwise idle server, CPU latency was very predictable. In our reading, that makes capacity planning simpler than on a shared GPU.

Why batching does not help on CPU

Up to 10 questions per call, each extra question costs about 600 ms on the English and typed-decisions checkpoints and about 185 ms on multilingual. At 50 questions the cost per question rises by another 15 to 20%. In other words, ten questions in one call take roughly as long as ten separate calls.

GPUs behave differently:

hardware	1 question	50 questions	cost per extra question
Tesla T4, `laya`	39.5 ms	771.3 ms	about 15 ms
NVIDIA GB10, `laya-typed-decisions` (over HTTP)	100.2 ms	443.1 ms	about 7 ms
EPYC 4-core CPU, `laya`	580 ms	35,969 ms	about 600 ms up to 10 questions, more beyond

On the GB10, roughly 93 ms of every call is fixed overhead, so packing questions into one call is where the speedup is. On CPU there is almost no fixed overhead to amortize, so the design choice is simpler: ask only the questions you need.

Thread settings: the easiest 12x

The upstream project also reports a laptop measurement (Ryzen 9 6900HX, WSL2) that is worth copying before anything else. With torch's default thread settings on a busy host (10 intra-op and 5 inter-op threads on 10 vCPUs), a three-question call over HTTP took 9,396 ms at p50. Setting two lines brought it to 783 ms, about 12x faster with no code change:

import torch

torch.set_num_threads(8)          # about the number of physical cores
torch.set_num_interop_threads(1)  # one forward pass per call: nothing to overlap

On the same laptop, one question in-process took 910 ms with 1 thread, 374 ms with 4, 329 ms with 8, and got worse again at 388 ms when every vCPU was used. The upstream advice is to use the physical core count plus a little, not one thread per vCPU, because SMT siblings contend. If you run the self-hosted server, LAYA_THREADS caps the same setting.

Cold starts and memory

A cold load took 4.4 s for the English checkpoint and 2.5 s for multilingual on the EPYC machine. These figures depend on the OS file cache, so treat them as approximate. Two practical consequences:

Preload the checkpoints you serve (Router(preload=True)) instead of loading on the first request.
Budget memory. The benchmark script peaked at 9.3 GiB with up to five checkpoints loaded at once. Serving fewer checkpoints should need less, but the upstream run does not report a per-checkpoint figure, so measure on your own machine.

When is a CPU enough?

This table is our interpretation of the numbers above, not a benchmark result. It assumes one question per call:

your situation	CPU is fine?
Background jobs, queues, nightly batch classification	Yes. Latency rarely matters; cost does.
Interactive routing or triage in non-English text	Usually. `laya-multilingual` at about 0.2 s is fast enough for many UIs.
Interactive English decisions with several questions per request	Borderline. Five questions on the English checkpoint take about 3 s.
High-throughput, low-latency serving	Use a GPU. A T4 answers one question in 32.8 to 39.5 ms.

These are single-process, in-process numbers. Real deployments add HTTP, queuing, and concurrency effects, so benchmark your own hardware with your own questions before committing. The upstream script is research/scripts/bench_latency.py.

Sources

Laya BENCHMARKS.md: server CPU, laptop CPU, T4, and GB10 sections
Raw EPYC results

Laya benchmarks: every result with its checkpoint, hardware, and test conditions
Self-host Laya: run Laya as an HTTP API, including LAYA_THREADS
Laya models: choose between the English, multilingual, and typed-decisions checkpoints

Last verified: September 25, 2026.