How to Fine-Tune Laya: RLCD Notebook, Calibration & Custom Heads

Fine-tuning is where most of Laya's value comes from. The upstream project says so directly: on its typed-decisions benchmark the base checkpoints score near chance zero-shot, while the fine-tuned checkpoint reaches 0.766 on the same 2,000 decisions. Treat Laya as a fast base to specialise, not a zero-shot decision engine.

This guide explains the official fine-tuning path, what the numbers mean, and how to avoid the calibration mistakes the upstream project has already documented.

Primary sources: Laya on GitHub and the laya-typed-decisions model card.

Why fine-tune at all?

Upstream results on the typed-decisions benchmark (400 cases, 2,000 decisions):

The base checkpoints sit below the majority-class baseline on this benchmark. All of the capability comes from fine-tuning on the benchmark's own 1,200-case training split. If your workflow is not one of those four synthetic workflows (invoice processing, security incidents, customer service, agent-trace observability), expect to fine-tune on your own data too.

What you need

• Labeled decisions from your domain. A state (ticket, email, JSON document) plus the correct answer for each typed question you care about.
• A GPU. The official notebook targets Kaggle's free 2x T4. Upstream reports roughly 4–5 hours for 4 epochs over ~30k questions.
• A held-out evaluation set that the model never trains on, including a slice reserved for calibration.
• Python 3.10+ and pip install laya (see Install Laya).

The quickest path: laya-train (0.3.28+)

Since 0.3.28 the package ships the same training loop as a command. From a CSV with one labelled decision per row:

pip install -U laya
laya-train --data tickets.csv --text-column body --label-column department \
           --base english --out ./tickets-checkpoint --eval held_out.csv

It turns the label column into a choice question, fine-tunes on one device (CUDA, MPS or CPU), fits calibration temperatures on a slice held out before training and saves a checkpoint laya.load opens, with the question beside it in questions.json. With --eval, it prints accuracy, ECE and Brier score before and after fine-tuning on your held-out file and writes them to train_report.json. A JSONL file with several questions of any type per case works too.

import json, laya

agent = laya.load("./tickets-checkpoint")
questions = json.load(open("./tickets-checkpoint/questions.json"))
print(agent.predict("We were billed twice for March.", questions)["answers"])

Useful flags: --dry-run reports how many rows were read, which questions were skipped and how many optimizer updates the run will make, without training; --freeze-encoder trains only the decision head, which fits a small GPU or a CPU but gains much less; --target-error (default 0.10) fits abstention thresholds for min_confidence; --loss soft-ce and --shuffle-options are the alternatives upstream benchmarked against the default RLCD loss. If the run warns that calibration was not fitted (fewer than 10 calibration items for a question type), do not gate on min_confidence with that checkpoint. Full options: upstream fine-tuning docs.

The official notebook

The upstream repository ships laya_finetune_typed_decisions_2xT4_kaggle.ipynb. It runs the whole loop:

1. Build the dataset of states and typed questions.
2. Train with RLCD (proper-scoring-rule rewards, GRPO-style policy gradient).
3. Fit calibration temperatures.
4. Evaluate.
5. Push the result to the Hugging Face Hub.

No two GPUs? laya-train above is the simplest route. Since 0.3.22, upstream also ships finetune_single_device.py, which runs the same training loop on one GPU or on CPU for any checkpoint, including laya-multilingual. On Apple Silicon, use laya_finetune_typed_decisions_mps.py.

The notebook enables gradient checkpointing on both the encoder and the decision head to fit in T4 memory. In a custom training loop, model.head_checkpointing = True enables activation checkpointing for the decision-head layers; the encoder's gradient checkpointing is enabled separately.

How RLCD training works

According to the model card, the policy reports a probability distribution over options, exploration adds zero-mean Gaussian noise to the logits, and the reward is a strictly proper scoring rule (log + spherical, plus ranked probability score for ordinal score questions). Under a strictly proper scoring rule, expected reward is maximised only by reporting honest probabilities. Updates use REINFORCE with a group-mean baseline, alongside a soft cross-entropy term against a teacher's distributions.

The practical consequence: Laya is trained to produce probabilities you can threshold on — but only after you check calibration on your own data.

Calibration: the step people skip

Upstream is explicit that the shipped checkpoints are over-confident:

• Refitting one temperature per (question type, option count) on held-out data moves mean ECE from 0.466 to 0.081 on laya and 0.314 to 0.106 on laya-multilingual.
• laya-multilingual ships with no fitted temperatures at all.
• The published laya-typed-decisions temperatures were fitted on a slice of its own training items, which is why they sit close to 1.0 (issue #186). The notebook now holds that slice out of training.

The notebook fits one temperature per type (choice, score, noul) and removes inherited temperature_by_options from the exported config, because stale bucket values take precedence at inference and silently mask a new fit.

Two rules follow:

1. Never fit calibration on training data. Use a held-out slice.
2. Evaluate on a separate test set before claiming an improvement. The notebook fixes configuration persistence; it does not by itself prove better accuracy or calibration.

Since 0.3.12, lang_temperatures= lets you apply your own per-language temperatures. Nothing ships with values, so you have to fit them.

Designing a fine-tuning dataset

These recommendations are our synthesis of the upstream limits, not upstream benchmark results:

• Mirror production questions exactly. Use the same instructions, option keys and descriptions you will serve.
• Keep choice questions under ~20 options. Options share a fixed head budget (192 tokens on laya, 256 on the other checkpoints). For larger label sets, split into coarse and fine questions or use predict_shortlist.
• Avoid boolean-word labels in choice questions (true/false, yes/no). Use semantic or opaque keys such as A/B.
• Include hard negatives and ambiguous cases so the model learns when not to be confident.
• Match language to checkpoint. Start from laya for English and laya-multilingual for other languages.

A worked example from the community

Upstream links a complete specialisation for a browser agent (write-up). It ran on a single 16 GB GPU with no paid API. Reported results: element top-1 among ~45 candidates went from 0.10 zero-shot to 0.66, and real-task success from 0% to 62%, at 17–23 ms per step. Weights and pipeline code are at cklxx/laya-browser.

A lighter alternative: train a head on the frozen encoder

If full RLCD fine-tuning is more than you need, the community project stuntd runs Laya behind the Jev API and trains a head per decision on the frozen encoder from your own labeled rows, with a calibrated confidence threshold. Its author reports a 12-label intent task going from 89.5% zero-shot to 100% trained. Those are the project's own figures on its own task.

After fine-tuning

• Load your checkpoint with laya.load("your-org/your-checkpoint"), or register it with a Router via router.attach(...).
• Gate automation on confidence and send low-confidence cases to a human or a larger model (see Laya Python guide).
• Upstream notes that action.act_probability carries no usable signal yet (#185); gate on confidence instead.
• Re-check thresholds whenever you change checkpoint, batch settings or precision.

Related

• What is RLCD?: the calibrated-decision training behind Laya and Jev
• Laya models compared
• laya-typed-decisions checkpoint
• Laya benchmarks and their conditions
• Self-host Laya as an HTTP API

Last verified: September 24, 2026.