Published on

Efficient Streaming Language Models with Attention Sinks

  • avatar

Efficient Streaming Language Models with Attention Sinks



Deploying Large Language Models (LLMs) in streaming applications, such as multi-round dialogue, where long interactions are expected, is a challenging task. Two major challenges arise in this context. Firstly, caching previous tokens' Key and Value states (KV) during the decoding stage consumes extensive memory. Secondly, popular LLMs struggle to generalize to longer texts than the training sequence length.

To address these challenges, the authors of the paper "Efficient Streaming Language Models with Attention Sinks" propose a framework called StreamingLLM. This framework enables LLMs trained with a finite length attention window to generalize to infinite sequence length without any fine-tuning.


The authors observe an interesting phenomenon called attention sink. They find that keeping the KV of initial tokens, even if they are not semantically important, largely recovers the performance of window attention. This is because there are strong attention scores towards initial tokens as a "sink". Based on this observation, the authors introduce StreamingLLM, an efficient framework that leverages attention sinks to enable LLMs to perform stable and efficient language modeling with up to 4 million tokens and more.

The authors also discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. This approach outperforms the sliding window recomputation baseline by up to 22.2x speedup in streaming settings.

How Does It Work?

The StreamingLLM framework is implemented using Python and relies on the PyTorch library. To set up the environment, you can use the provided code snippet:

conda create -yn streaming python=3.8
conda activate streaming

pip install torch torchvision torchaudio
pip install transformers accelerate datasets evaluate wandb scikit-learn scipy

python develop

Once the environment is set up, you can run the Streaming Llama Chatbot using the following command:

CUDA_VISIBLE_DEVICES=0 python examples/ --enable_streaming

Benefits and Use Cases

The StreamingLLM framework has several benefits and use cases. By enabling LLMs to handle infinite-length inputs without sacrificing efficiency and performance, it opens up new possibilities for streaming applications. Some potential use cases include:

  1. Multi-round dialogue systems: StreamingLLM can be used to build chatbots or virtual assistants that engage in long conversations with users.

  2. Real-time language translation: StreamingLLM can be applied to language translation systems, allowing them to process and translate long texts in real-time.

  3. Continuous speech recognition: StreamingLLM can be used to develop speech recognition systems that can transcribe long audio streams without interruption.

  4. Text generation: StreamingLLM can generate coherent and context-aware text in real-time, making it useful for applications such as chatbots, content generation, and storytelling.

Future Directions

The authors of the paper mention several future directions for their research. They plan to release the code and data in a specific order, including the core code of StreamingLLM, perplexity evaluation code, the Streaming Llama Chatbot demo, and the StreamEval dataset and evaluation code. By making these resources available, they hope to facilitate further research and development in the field of efficient streaming language models.


The StreamingLLM framework presented in the paper "Efficient Streaming Language Models with Attention Sinks" offers a solution to the challenges of deploying LLMs in streaming applications. By leveraging attention sinks and introducing a dedicated attention sink token, StreamingLLM enables LLMs to handle infinite-length inputs efficiently and effectively. The framework has various use cases in multi-round dialogue, language translation, speech recognition, and text generation. With the release of code and data, the authors aim to foster further advancements in the field of efficient streaming language models.

To learn more about StreamingLLM, you can read the full paper here and access the code repository here.