A Predictive Law for On-Policy Self-Distillation From World Feedback

Paper: https://arxiv.org/abs/2605.30070

Summary

On-policy self-distillation (OPSD) trains a student policy to match a self-teacher - the same model conditioned on privileged context (e.g. environment feedback, peer solutions, expert hints). The choice of privileged context drives OPSD's effectiveness, but its impact on final performance has been hard to anticipate without running full training.

We show that the initial student–self-teacher accuracy gap, measured before any training, is a strong linear predictor of the final student improvement after OPSD post-training. The relationship holds across:

• Six qualitatively different privileged-context constructions (none, expert preamble, environment feedback, LLM peer hints, own solution + feedback, peer solution + feedback)
• Two model families (Qwen3, Olmo 3)
• Four model sizes (Qwen3 0.6B / 1.7B / 4B / 8B)

The practical consequence is that you can screen OPSD configurations cheaply before launching a full training run and predict gains at larger scales from cheaper measurements.

Main result

For both Qwen3-8B and Olmo-3-7B-Instruct, the initial student–self-teacher gap linearly predicts final student improvement on LiveCodeBench v6 (Δ mean@4). OLS fits: Qwen3-8B R² = 0.949, Olmo-3-7B R² = 0.996. Each point is one privileged-context configuration averaged over 3 seeds.

Training dynamics

During 50 steps of OPSD on Qwen3-8B with the Peer Solution + Feedback context, the student's mean@4 (purple) steadily converges toward the self-teacher's (blue). The self-teacher also gains slightly over training, raising the ceiling.

Scaling

The same linear law holds across Qwen3-{0.6B, 1.7B, 4B, 8B} with the Peer Solution + Feedback context fixed (OLS R² = 0.977), suggesting the gap is a scale-invariant predictor of OPSD improvement.

Install

Environment is managed with uv. Python 3.12.

uv sync                        # creates .venv from pyproject.toml + uv.lock
source .venv/bin/activate
echo "WANDB_API_KEY=..." > .env  # sourced by run.sbatch

Direnv users can cd in and the .envrc does the venv activation automatically.

Run

Default: 50-step OPSD on Qwen3-8B with the all (Peer Solution + Feedback) context.

sbatch run.sbatch

Knobs in run.sbatch:

• CONSTRUCTION_METHOD (line 52) — one of all, feedback_only, correct_solution_processed, self_correct_solution_with_feedback, none, static_teacher1, etc.
• The model arg on the trailing bash ./experiments/rich_feedback/run_sdpo_lcb.sh ... line

LiveCodeBench v6 train/test parquets are produced automatically by data/prepare_lcb.sh on first run (downloads from HuggingFace).

Tested on

Item	Value
Hardware (paper experiments)	NVIDIA B200 GPUs (≈720 B200-hours total)
Cluster (this repo)	SLURM, 4 GPUs / run
OS	Linux x86_64
Python	3.12
CUDA toolkit	12.8 (PyTorch cu128 wheels)
PyTorch	2.9.1
vLLM	0.15.1
flash-attn	2.8.3
Container image	docker://the41/cuda-dev

Attribution

Built on top of SDPO (Hübotter et al., 2026) and verl (Sheng et al., 2025).