Researchers have found a way to spend 15% less calculations on LLM training without losing quality. At EMNLP 2026, an A* conference currently taking place in Hungary, researchers from T-Tech presented the LIMe method, which allows training language models at about 15% lower cost in terms of calculations without a decrease in quality. In transformers, each layer relies only on the hidden state of the previous one, and all context must pass through it. The authors show that this creates pressure towards the collapse of beliefs. That is, tokens become too similar to each other, the differences between them are lost, and together with them the quality is lost. LIMe (Layer Integrated Memory) is a lightweight extension of the attention mechanism. It uses key-value stores, already stored for each layer, and adds trainable routing weights to them, separate for each head and each layer. With their help, the head of attention can take representations not only from the previous layer, but also from the previous layer, and mix them. The size of the hidden state is not increased, and calculations already performed are reused. In model experiments with about a billion parameters, LIMe achieved the same training error as the baseline llama at a 15% lower cost. In measures of language modeling and multistage thinking, the method outperformed other approaches. The most notable gains were made in synthetic calculus problems and in architectures with a large number of layers. The authors also note that LIMe has a higher entropy of value vectors and a better separation of tokens, meaning that the main problem is really reduced. And from the learned routing weights it is clear that the model systematically reuses both the local and the long-range features. Article: Full implementation of the authors.

Researchers have found a way to spend 15% less calculations on LLM training without losing quality. At EMNLP 2026, an A* conference currently taking place in Hungary, researchers from T-Tech presented the LIMe method, which allows training language models at about 15% lower cost in terms of calculations without a decrease in quality. In transformers, each layer relies only on the hidden state of the previous one, and all context must pass through it. The authors show that this creates pressure towards the collapse of beliefs. That is, tokens become too similar to each other, the differences between them are lost, and together with them the quality is lost. LIMe (Layer Integrated Memory) is a lightweight extension of the attention mechanism. It uses key-value stores, already stored for each layer, and adds trainable routing weights to them, separate for each head and each layer. With their help, the head of attention can take representations not only from the previous layer, but also from the previous layer, and mix them. The size of the hidden state is not increased, and calculations already performed are reused. In model experiments with about a billion parameters, LIMe achieved the same training error as the baseline llama at a 15% lower cost. In measures of language modeling and multistage thinking, the method outperformed other approaches. The most notable gains were made in synthetic calculus problems and in architectures with a large number of layers. The authors also note that LIMe has a higher entropy of value vectors and a better separation of tokens, meaning that the main problem is really reduced. And from the learned routing weights it is clear that the model systematically reuses both the local and the long-range features. Article: Full implementation of the authors.