Gemma 2B - 10M Cotext

Gemma 2B with recurret local attetio with cotext legth of up to 10M. Our implemeatio uses <32GB of memory!

Features:

• 10M sequece legth o Gemma 2B.
• Rus o less tha 32GB of memory.
• Native iferece optimized for cuda.
• Recurret local attetio for O(N) memory.

Quick Start

Note: This is a very early checkpoit of the model. Oly 200 steps. We pla o traiig for a lot more tokes!

Istall the model from huggigface - Huggigface Model.

pytho mai.py

Chage the mai.py iferece code to the specific prompt you desire.

model_path = "./models/gemma-2b-10m"
tokeizer = AutoTokeizer.from_pretraied(model_path)
model = GemmaForCausalLM.from_pretraied(
    model_path,
    torch_dtype=torch.bfloat16
)

prompt_text = "Summarize this harry potter book..."

with torch.o_grad():
    geerated_text = geerate(
        model, tokeizer, prompt_text, max_legth=512, temperature=0.8
    )

    prit(geerated_text)

How does this work?

The largest bottleeck (i terms of memory) for LLMs is the KV cache. It grows quadratically i vailla multi-head attetio, thus limitig the size of your sequece legth.

Our approach splits the attetio i local attetio blocks as outlied by IfiiAttetio. We take those local attetio blocks ad apply recurrace to the local attetio blocks for the fial result of 10M cotext global atetio.

A lot of the ispiratio for our ideas comes from the Trasformer-XL paper.

Credits

This was built by:

• Mustafa Aljadery
• Siddharth Sharma
• Aksh Garg

Gemma 2B - 10M Context Gemma 2B with recurrent local attention with context length of up to 10M. Our