Community runtime

Laya MLX for Apple Silicon

Community-native MLX runtime for fast local typed decisions on M-series Macs without PyTorch.

Apple SiliconLocal inferenceLow latencyMac developer tools
Quickstart
pip install laya-mlx
Source
mizorewww/laya-mlx

Apple Silicon only. Check the runtime repository for converted checkpoint compatibility and version notes.

laya-mlx is an independent MLX port of Laya: the encoder, decision Transformer, scoring head and action head all run in Apple's MLX, with no PyTorch, Transformers runtime or cloud API at inference time. It keeps the original weights, prompt formatting, calibration and output schema.

Requirements

  • Apple Silicon Mac, macOS 14+, Python 3.11+.
  • The first load downloads the checkpoint; after that, inference is fully local.

Measured speed (project README, M3 Max, FP16)

End-to-endLaya 421MMultilingual 322M
One short question, P5013.42 ms7.39 ms
One short question, P9513.92 ms7.79 ms
50-question throughput146.8 q/s395.0 q/s
Peak MLX memory, one question943.6 MiB687.6 MiB

Timing includes prompt preparation, tokenization, inference, calibration and formatting; model loading is excluded. Different hardware, lengths and question counts change these numbers.

Port fidelity: all three checkpoints matched the upstream answer on 63/63 validation questions in both FP32 and FP16 (378/378 comparisons). The README notes this measures fidelity on those fixtures, not accuracy in general.

Usage

import laya_mlx as laya

agent = laya.load("aac6fef/laya-mlx", dtype="float16")
result = agent.predict(
    "I was billed twice. Please refund the duplicate today.",
    {
        "department": {"type": "choice",
                       "instructions": "Which team should handle this request?",
                       "criteria": {"billing": "invoices, payments, refunds",
                                    "technical": "bugs and outages"}},
        "refund": {"type": "noul", "instructions": "Does the customer ask for money back?"},
    },
)

Pre-converted FP16 checkpoints: aac6fef/laya-mlx, aac6fef/laya-multilingual-mlx, aac6fef/laya-typed-decisions-mlx. The original IDs (convaiinnovations/laya, with subfolder=) also load.

Features worth knowing

  • Router with upstream's language routing and presets, e.g. Router(dtype="float16", max_loaded=2); Chinese text routes to the multilingual checkpoint.
  • Precision: FP16 by default; dtype="float32" for closer numerical agreement with upstream.
  • Repeated workloads: opt into compile=True, pad_to_multiple=16 and cache_prompts=True.
  • Large label sets: predict_shortlist keeps the top-k labels by embedding similarity before one forward pass.
  • CLI and conversion: laya-mlx predict ... and laya-mlx convert --model convaiinnovations/laya --dtype float16.
  • Demo: pip install 'laya-mlx[demo]' then laya-snake plays Snake with a Laya decision on every move.

Limits

  • Apple Silicon only.
  • Inference and conversion only; RLCD training and fine-tuning stay in the upstream Python project.
  • The port preserves the model's own limits: the English checkpoint is not a substitute for the multilingual one, and confidence does not guarantee accuracy.

Also on Apple Silicon: the MPS runtime (PyTorch on the Mac GPU, low-memory modes) and Arbiter, which can use this MLX port as an engine.

Source: mizorewww/laya-mlx · Apache 2.0 · independent port, not an official Convai Innovations release.

Last verified against the project README: September 24, 2026.

Browse recipesView benchmarksExplore projects