SAM in mid-training may reduce forgetting during fine-tuning
A May 8, 2026 post says applying SAM during the final 10% of mid-training reduced forgetting by over 35% after fine-tuning or quantization.
Source: SAM no mid-training para reduzir o esquecimento no fine-tuning (x.com). Text prepared with AI from this source.
What happened and what to do
In a post published on May 8, 2026, the proposed approach controls sharpness with SAM during roughly the final 10% of mid-training. The post says this may reduce forgetting by over 35% after fine-tuning or quantization, even if base-model quality declines. It also suggests testing learning rates up to about ten times higher.
A company training or adapting models can evaluate the technique on its own data and tasks, comparing retention, quality, and cost across configurations. An experiment pipeline can record parameters and results and display comparisons in a dashboard, supporting decisions about training and quantization.
How the consultancy can help
Wendelmaques can diagnose the training process and forgetting risks, define an evaluation scope, and implement experimentation, monitoring, and analysis pipelines. Ongoing operation can be included according to the agreed scope.
Next step
Send a brief description of your model training or adaptation case to receive a scoped proposal.
Let's assess your project
I reply with scope and a proposal.