Serving layer

Laya Local Serving with Arbiter

Serve Laya behind a Jev-compatible HTTP API with checkpoint routing, batching, metrics, a Playground, and NVIDIA / Apple Silicon recipes.

Local APIJev-compatible clientsNVIDIA GPUApple SiliconAgent integrations
Quickstart
git clone https://github.com/0xBakeer/arbiter.git
cd arbiter
./run.sh setup
./run.sh serve
Source
0xBakeer/arbiter

This is a community serving layer rather than the upstream Laya package. Validate its defaults against your deployment requirements.

Arbiter is a small HTTP serving layer for typed-decision models. It speaks TypeSafe's Jev API (POST /v1/systemone), so a client written against Jev works by changing the base URL, and it keeps all three Laya checkpoints resident with automatic routing between them.

Install

Python 3.12+ and one of two lanes: an NVIDIA GPU (CUDA 13.0 wheels, aarch64 and x86_64) or Apple Silicon (PyTorch MPS, with an opt-in MLX engine).

git clone https://github.com/0xBakeer/arbiter.git
cd arbiter
./run.sh setup      # venv, torch, dependencies and the three checkpoints (2.3 GB)
./run.sh serve      # http://localhost:8010

./run.sh status, smoke, examples, bench, test and stop cover the rest. The server also hosts a playground page at / and six browser games at /showcase/.

Measured speed (project README, end-to-end over HTTP)

1 question1050
NVIDIA GB10, LLM idle20.9 ms40.0 ms152.4 ms
NVIDIA GB10, next to a busy LLM326.6 ms258.1 ms381.6 ms
Apple M2 Max, fp3230.3 ms107.4 ms462.3 ms
Apple M2 Max, laya_mlx engine22.7 ms102.7 ms448.4 ms

All three checkpoints use about 5 GB of GPU memory (about 2.9 GB in bf16), small enough to sit next to a large language model on the same card, though calls slow down roughly 10–15x while that LLM is decoding.

Routing

model you sendWhat runs
"laya", "auto", "jev-latest" or nothingthe router picks by script and language
"english" / "laya-english"English, 512 tokens
"multilingual" / "laya-multilingual"100+ languages, 1,024 tokens
"typed-decisions"the fine-tuned checkpoint; never chosen automatically

Configuration highlights

  • ARBITER_API_KEY requires Authorization: Bearer <key>.
  • ARBITER_MODELS loads a subset (English alone is about 1.9 GB).
  • ARBITER_DTYPE=bf16 is 15–35% faster on CUDA but moves probabilities by up to 2.1e-2, so the default stays autocast.
  • ARBITER_MODE=graphs (CUDA only) is about 20% faster for single small calls, slower for large ones.
  • /metrics exposes Prometheus metrics; /readyz reports when checkpoints are loaded and warm.

Integrations and examples

Nine runnable examples cover support triage, email triage, agent shell-command guarding, pull-request risk gating, alert triage, model routing, RAG relevance, moderation and invoice fields. An MCP server and a Claude Code plugin (a PreToolUse hook that judges every Bash command) are included; see Laya integrations.

When to choose it

Pick Arbiter over upstream's laya-serve when you want cross-request micro-batching, Prometheus metrics, a playground and measured per-machine recipes. The model's own limits still apply: the base checkpoints are weak zero-shot on custom workflows and probabilities need calibration.

Source: 0xBakeer/arbiter · MIT (model weights Apache 2.0).

Last verified against the project README: September 24, 2026.

Browse recipesView benchmarksExplore projects