How to Fine-Tune Laya: RLCD Notebook, Calibration & Custom Heads

Fine-tuning is where most of Laya's value comes from. The upstream project says so directly: on its typed-decisions benchmark the base checkpoints score near chance zero-shot, while the fine-tuned checkpoint reaches 0.766 on the same 2,000 decisions. Treat Laya as a fast base to specialise, not a zero-shot decision engine.

This guide explains the official fine-tuning path, what the numbers mean, and how to avoid the calibration mistakes the upstream project has already documented.

Primary sources: Laya on GitHub and the laya-typed-decisions model card.

Why fine-tune at all?

Upstream results on the typed-decisions benchmark (400 cases, 2,000 decisions):

Checkpoint	Accuracy	Brier (lower is better)	Score MAE
`laya` (not fine-tuned)	0.362	0.316	0.694
`laya-multilingual` (not fine-tuned)	0.342–0.352	0.439–0.463	0.687–0.760
`laya-typed-decisions` (fine-tuned)	0.766	0.062	0.242
Per-question majority class	0.461
Random guess	0.318

The base checkpoints sit below the majority-class baseline on this benchmark. All of the capability comes from fine-tuning on the benchmark's own 1,200-case training split. If your workflow is not one of those four synthetic workflows (invoice processing, security incidents, customer service, agent-trace observability), expect to fine-tune on your own data too.

What you need

Labeled decisions from your domain. A state (ticket, email, JSON document) plus the correct answer for each typed question you care about.
A GPU. The official notebook targets Kaggle's free 2x T4. Upstream reports roughly 4–5 hours for 4 epochs over ~30k questions.
A held-out evaluation set that the model never trains on, including a slice reserved for calibration.
Python 3.10+ and pip install laya (see Install Laya).

The official notebook

The upstream repository ships laya_finetune_typed_decisions_2xT4_kaggle.ipynb. It runs the whole loop:

Build the dataset of states and typed questions.
Train with RLCD (proper-scoring-rule rewards, GRPO-style policy gradient).
Fit calibration temperatures.
Evaluate.
Push the result to the Hugging Face Hub.

The notebook enables gradient checkpointing on both the encoder and the decision head to fit in T4 memory. In a custom training loop, model.head_checkpointing = True enables activation checkpointing for the decision-head layers; the encoder's gradient checkpointing is enabled separately.

How RLCD training works

According to the model card, the policy reports a probability distribution over options, exploration adds zero-mean Gaussian noise to the logits, and the reward is a strictly proper scoring rule (log + spherical, plus ranked probability score for ordinal score questions). Under a strictly proper scoring rule, expected reward is maximised only by reporting honest probabilities. Updates use REINFORCE with a group-mean baseline, alongside a soft cross-entropy term against a teacher's distributions.

The practical consequence: Laya is trained to produce probabilities you can threshold on — but only after you check calibration on your own data.

Calibration: the step people skip

Upstream is explicit that the shipped checkpoints are over-confident:

Refitting one temperature per (question type, option count) on held-out data moves mean ECE from 0.466 to 0.081 on laya and 0.314 to 0.106 on laya-multilingual.
laya-multilingual ships with no fitted temperatures at all.
The published laya-typed-decisions temperatures were fitted on a slice of its own training items, which is why they sit close to 1.0 (issue #186). The notebook now holds that slice out of training.

The notebook fits one temperature per type (choice, score, noul) and removes inherited temperature_by_options from the exported config, because stale bucket values take precedence at inference and silently mask a new fit.

Two rules follow:

Never fit calibration on training data. Use a held-out slice.
Evaluate on a separate test set before claiming an improvement. The notebook fixes configuration persistence; it does not by itself prove better accuracy or calibration.

Since 0.3.12, lang_temperatures= lets you apply your own per-language temperatures. Nothing ships with values, so you have to fit them.

Designing a fine-tuning dataset

These recommendations are our synthesis of the upstream limits, not upstream benchmark results:

Mirror production questions exactly. Use the same instructions, option keys and descriptions you will serve.
Keep choice questions under ~20 options. Options share a fixed head budget (192 tokens on laya, 256 on the other checkpoints). For larger label sets, split into coarse and fine questions or use predict_shortlist.
Avoid boolean-word labels in choice questions (true/false, yes/no). Use semantic or opaque keys such as A/B.
Include hard negatives and ambiguous cases so the model learns when not to be confident.
Match language to checkpoint. Start from laya for English and laya-multilingual for other languages.

A worked example from the community

Upstream links a complete specialisation for a browser agent (write-up). It ran on a single 16 GB GPU with no paid API. Reported results: element top-1 among ~45 candidates went from 0.10 zero-shot to 0.66, and real-task success from 0% to 62%, at 17–23 ms per step. Weights and pipeline code are at cklxx/laya-browser.

A lighter alternative: train a head on the frozen encoder

If full RLCD fine-tuning is more than you need, the community project stuntd runs Laya behind the Jev API and trains a head per decision on the frozen encoder from your own labeled rows, with a calibrated confidence threshold. Its author reports a 12-label intent task going from 89.5% zero-shot to 100% trained. Those are the project's own figures on its own task.

After fine-tuning

Load your checkpoint with laya.load("your-org/your-checkpoint"), or register it with a Router via router.attach(...).
Gate automation on confidence and send low-confidence cases to a human or a larger model (see Laya Python guide).
Upstream notes that action.act_probability carries no usable signal yet (#185); gate on confidence instead.
Re-check thresholds whenever you change checkpoint, batch settings or precision.

Last verified: September 24, 2026.