LoRA+ proposes distinct learning rates for adapter matrices
A September 24, 2026 article examines how using the same learning rate for matrices A and B can hinder feature learning in wide models, and proposes different rates for fine-tuning.
Source: LoRA+ usa taxas de aprendizado diferentes para matrizes adapter (arxiv.org). Text prepared with AI from this source.
What happened and what to do
Published on September 24, 2026, the article examines LoRA use in wide models. It says applying the same learning rate to matrices A and B can hinder feature learning, and proposes using different rates for each matrix during fine-tuning.\n\nA company adapting models can test this configuration on its own datasets and tasks. A traceable training process can record parameters and results, compare configurations, and monitor metrics defined for each case, supporting a decision without assuming the change improves every model or task.
How the consultancy can help
Wendelmaques can diagnose your fine-tuning process, define a comparative test with evaluation criteria, and implement collection, monitoring, and operation of the training workflow.
Next step
Send a short description of your model, task, and fine-tuning process to receive a scoped proposal for your case.
Let's assess your project
I reply with scope and a proposal.