Self-Host Laya: Jev-Compatible HTTP API, CLI, Docker & MCP Server

The upstream laya Python package can do more than run inside your own script. It ships a self-hosted HTTP server that speaks the same wire protocol as TypeSafe's hosted Jev API, a command-line tool for quick tests, and an optional MCP server for agent clients.

This guide collects those options in one place. Primary source: Laya on GitHub.

Just want to call Laya over HTTP without running it? See the Laya API guide for the request format and hosted options.

Option 1: the Jev-compatible HTTP server (laya-serve)

laya.serve exposes the Router over HTTP at POST /v1/systemone. Upstream states that Laya's answer payload is already schema-identical to what Jev returns (choice / score / noul answers plus an {input_tokens, output_tokens} usage block), so an existing Jev client only needs its base URL changed.

pip install "laya[serve]"                     # adds fastapi, uvicorn, python-multipart
LAYA_DEVICE=cuda LAYA_PRELOAD=1 laya-serve    # binds 0.0.0.0:8000, preloads all 3 checkpoints

Send a request:

curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "state": {"body": "billed twice, refund please or we cancel"},
  "questions": {"dept": {"type": "choice", "instructions": "which team?",
                "criteria": {"billing": "refunds", "tech": "bugs"}}}
}'

Since 0.3.22 the server also takes POST /v1/systemone/batch: a states list (up to 64) with one shared questions object, returning one result per state plus total_usage. The request body can also set task, lang, lang_guess and min_confidence, which older versions silently ignored.

Configuration

A client's model field is honoured when it names a Laya checkpoint (english, multilingual or typed-decisions); otherwise the router selects one by script and language.

Security defaults

Since 0.3.11 the server uses timing-safe API key checks, enforces request size and question limits (HTTP 413), returns 400 for malformed JSON and avoids leaking paths in errors. 0.3.12 also enforces the body-size limit on chunked uploads. If you expose the server beyond localhost, set LAYA_API_KEY and put it behind your usual TLS proxy.

Option 2: Docker

Upstream maintains a Docker Compose quickstart in docs/docker.md. It runs a sample request on CPU and keeps downloaded models between runs. The Compose file binds laya-serve to loopback by default and adds a healthcheck. Native ARM64 and DGX Spark container builds are also provided.

Option 3: NixOS

The repository is a Nix flake. On a machine with an NVIDIA GPU, nix run .#laya-serve builds and serves. For a NixOS host, import laya.nixosModules.default and enable services.laya-serve, which runs a hardened DynamicUser systemd unit, caches weights under /var/lib/laya-serve and reads an optional bearer token through LoadCredential.

Option 4: the local web GUI

For manual testing without writing a client, examples/server.py in the upstream repository is a self-contained FastAPI app with a request builder that renders choice / score / noul answers as bars, plus /predict and /predict/batch JSON endpoints:

pip install "laya[serve]"
python examples/server.py        # http://127.0.0.1:8000

--no-preload loads checkpoints lazily and --device cuda|cpu|mps pins the device.

Option 5: the laya command

Installing the package also installs a laya command:

laya "I was charged twice, please refund"            # routing decision only; offline, no download
laya "Refactor this service" --predict               # full answers (downloads the checkpoint once)
laya "Mein Konto wurde zweimal belastet" --lang de   # force a language
laya "My payment failed twice" --preset triage       # presets: triage, email, guard, moderation, router

Routing alone never downloads a checkpoint, so it returns in milliseconds.

Option 6: MCP server for agent clients

Laya can run as an MCP stdio server so MCP clients such as Claude Desktop or Cursor can call typed decisions as tools (laya_predict, laya_route, laya_preset, laya_status):

pip install "laya[mcp]"
laya-mcp-server

{
  "mcpServers": {
    "laya": {
      "command": "laya-mcp-server",
      "env": { "LAYA_DEVICE": "cpu" }
    }
  }
}

The MCP server preloads english,multilingual by default and leaves typed-decisions lazy. As with the SDK, use it for structured decisions, not open-ended Q&A.

Sizing: latency and memory

Upstream measurements with Router(preload=True): 32.8 ms per request on a T4 GPU and 193–464 ms on CPU. A lazy Router() keeps two checkpoints resident; max_loaded=1 reloads on every language switch, measured at a 7.4 s median on CPU and 10.3 s on a T4. For a server, preload what you serve.

On a plain 4-core server CPU (AMD EPYC 9R14, v0.3.20, no GPU), upstream measured 193 ms per question on laya-multilingual and about 580 ms on the English and typed-decisions checkpoints. Cost grows roughly linearly with the number of questions, so batching questions saves little on CPU. If you serve on CPU only, prefer the multilingual checkpoint and set LAYA_THREADS to the physical core count. See Laya benchmarks for the full numbers and Laya on CPU for thread settings and sizing.

Download sizes from the model card: about 808 MB for the English checkpoint and 647 MB for the multilingual one.

Community serving layers

If you want more than the upstream server, see Arbiter (checkpoint routing, batching, metrics and a Playground) or the macOS MPS runtime. The Runtimes overview compares all options.

Related

• Laya and Ollama: run Laya locally with ollaya
• Laya and Unsloth: a GUI Decision API
• Install Laya
• Laya vs Jev
• Laya integrations: MCP, browser agents and coding agents
• Fine-tune Laya

Last verified: September 24, 2026.