What attention sinks reveal about LLMs
Experiments on models from 410 million to 120 billion parameters link extreme activation norms of the initial token to compression valleys and attention sinks.
Source: Attention Sinks e Compression Valleys em LLMs (x.com). Text prepared with AI from this source.
What happened and what to do
In a paper published on October 9, 2025, experiments on models from 410 million to 120 billion parameters report that extreme activation norms for the beginning-of-sequence token in intermediate layers coincide with compression valleys and attention sinks. The authors propose computational phases in Transformers to describe how information mixing changes across layers.
For a company operating models internally, the practical takeaway is to instrument inference and compare activations and behavior across layers on relevant tasks. A controlled analysis can help clarify the model’s information flow and guide evaluation or optimization decisions, without assuming these phenomena produce gains on their own.
How the consultancy can help
Wendelmaques can diagnose an instrumentation and evaluation opportunity, define metrics and scope, and implement collection, analysis, and monitoring pipelines for models on the company’s infrastructure, as well as support their operation.
Next step
Send a short description of the model and the challenge your team wants to investigate to receive a scoped proposal.
Let's assess your project
I reply with scope and a proposal.