LLM 基础设施

使用 Laya 进行模型路由

在进入昂贵推理链路前，决定使用小模型、均衡模型还是 frontier model。

在 Playground 体验安装 Laya

使用场景

使用 typed choice 作为策略层，为不同请求选择下游模型。

State

Prove whether the following Rust unsafe block can cause undefined behavior and explain the exact aliasing rule involved.

Typed questions

choicemodel_tier

Which model tier should handle this request?

smallbalancedfrontier
noulneeds_reasoning

Does this request require deep technical reasoning?

预期方向

  • • 查看主要结构化结果是否符合预期
  • • 检查概率 / score 是否支持你的业务阈值
路由阈值属于业务策略，应使用真实流量验证 accuracy 与 calibration。

直接复制代码

Python
from laya import Router

router = Router(preload=True)
state = {"request": "Prove whether this Rust unsafe block can cause undefined behavior."}
questions = {
    "model_tier": {
        "type": "choice",
        "instructions": "Which model tier should handle this request?",
        "criteria": {"small": "simple extraction", "balanced": "normal reasoning", "frontier": "deep technical reasoning"}
    },
    "needs_reasoning": {"type": "noul", "instructions": "Does this request require deep technical reasoning?"}
}
print(router.predict(state, questions))
TypeScript / Node.js
import { Laya } from "@receptron/laya";
const laya = await Laya.load();
const result = await laya.systemOne(
  { request: "Prove whether this Rust unsafe block can cause undefined behavior." },
  {
    model_tier: { type: "choice", instructions: "Which model tier?", criteria: { small: "simple", balanced: "normal reasoning", frontier: "deep technical reasoning" } },
    needs_reasoning: { type: "noul", instructions: "Does this require deep technical reasoning?" },
  },
);
console.log(result.answers);
await laya.close();

继续阅读

Laya benchmarksRuntime options