On-Policy Self-Distillation for Reasoning in LLMs
A paper dated September 28, 2026 describes a training method that combines trajectories generated by the student with dense teacher supervision at each token.
Source: Self-Distillation On-Policy para Raciocínio em LLMs (arxiv.org). Text prepared with AI from this source.
What happened and what to do
The paper presents Self-Distilled Reasoner, an on-policy self-distillation approach for language models. The student generates the training trajectories, while a teacher provides dense token-level supervision. The proposal addresses the distribution mismatch between training and inference associated with off-policy distillation. The paper is dated September 28, 2026.
A company training models for reasoning tasks can evaluate this method with an experimentation pipeline: log trajectories, compare supervision strategies, and measure performance on its own evaluations. Implementation should include controls to reproduce experiments and monitor cost and quality, without assuming the technique is suitable before testing.
How the consultancy can help
Wendelmaques can diagnose the opportunity in the context of your models and data, define an evaluation scope, and implement and operate the training, experimentation, and monitoring pipeline.
Next step
Send a short description of your case to receive a scoped proposal for diagnosis and implementation.
Let's assess your project
I reply with scope and a proposal.