Model Dowload | Evaluatio Results | Model Architecture | API Platform | Licese | Citatio

Paper Lik?️

DeepSeek-V2: A Strog, Ecoomical, ad Efficiet Mixture-of-Experts Laguage Model

1. Itroductio

Today, we’re itroducig DeepSeek-V2, a strog Mixture-of-Experts (MoE) laguage model characterized by ecoomical traiig ad efficiet iferece. It comprises 236B total parameters, of which 21B are activated for each toke. Compared with DeepSeek 67B, DeepSeek-V2 achieves stroger performace, ad meawhile saves 42.5% of traiig costs, reduces the KV cache by 93.3%, ad boosts the maximum geeratio throughput to 5.76 times.

We pretraied DeepSeek-V2 o a diverse ad high-quality corpus comprisig 8.1 trillio tokes. This comprehesive pretraiig was followed by a process of Supervised Fie-Tuig (SFT) ad Reiforcemet Learig (RL) to fully uleash the model's capabilities. The evaluatio results validate the effectiveess of our approach as DeepSeek-V2 achieves remarkable performace o both stadard bechmarks ad ope-eded geeratio evaluatio.

2. Model Dowloads

Due to the costraits of HuggigFace, the ope-source code curretly experieces slower performace tha our iteral codebase whe ruig o GPUs with Huggigface. To facilitate the efficiet executio of our model, we offer a dedicated vllm solutio that optimizes performace for ruig our model effectively.

3. Evaluatio Results

Base Model

Stadard Bechmark

For more evaluatio details, such as few-shot settigs ad prompts, please check our paper.

Cotext Widow

Evaluatio results o the Needle I A Haystack (NIAH) tests. DeepSeek-V2 performs well across all cotext widow legths up to 128K.

Chat Model

Stadard Bechmark

Eglish Ope Eded Geeratio Evaluatio

We evaluate our model o AlpacaEval 2.0 ad MTBech, showig the competitive performace of DeepSeek-V2-Chat-RL o Eglish coversatio geeratio.

Chiese Ope Eded Geeratio Evaluatio

Aligbech (https://arxiv.org/abs/2311.18743)

Codig Bechmarks

We evaluate our model o LiveCodeBech (0901-0401), a bechmark desiged for live codig challeges. As illustrated, DeepSeek-V2 demostrates cosiderable proficiecy i LiveCodeBech, achievig a Pass@1 score that surpasses several other sophisticated models. This performace highlights the model's effectiveess i tacklig live codig tasks.

4. Model Architecture

DeepSeek-V2 adopts iovative architectures to guaratee ecoomical traiig ad efficiet iferece：

• For attetio, we desig MLA (Multi-head Latet Attetio), which utilizes low-rak key-value uio compressio to elimiate the bottleeck of iferece-time key-value cache, thus supportig efficiet iferece.
• For Feed-Forward Networks (FFNs), we adopt DeepSeekMoE architecture, a high-performace MoE architecture that eables traiig stroger models at lower costs.

5. Chat Website

You ca chat with the DeepSeek-V2 o DeepSeek's official website: chat.deepseek.com

6. API Platform

We also provide OpeAI-Compatible API at DeepSeek Platform: platform.deepseek.com. Sig up for over millios of free tokes. Ad you ca also pay-as-you-go at a ubeatable price.

7. How to ru locally

To utilize DeepSeek-V2 i BF16 format for iferece, 80GB*8 GPUs are required.

Iferece with Huggigface's Trasformers

You ca directly employ Huggigface's Trasformers for model iferece.

Text Completio

import torch
from trasformers import AutoTokeizer, AutoModelForCausalLM, GeeratioCofig

model_ame = "deepseek-ai/DeepSeek-V2"
tokeizer = AutoTokeizer.from_pretraied(model_ame, trust_remote_code=True)
# `max_memory` should be set based o your devices
max_memory = {i: "75GB" for i i rage(8)}
model = AutoModelForCausalLM.from_pretraied(model_path, trust_remote_code=True, device_map="auto", torch_dtype=torch.bfloat16, max_memory=max_memory)
model.geeratio_cofig = GeeratioCofig.from_pretraied(model_ame)
model.geeratio_cofig.pad_toke_id = model.geeratio_cofig.eos_toke_id

text = "A attetio fuctio ca be described as mappig a query ad a set of key-value pairs to a output, where the query, keys, values, ad output are all vectors. The output is"
iputs = tokeizer(text, retur_tesors="pt")
outputs = model.geerate(**iputs.to(model.device), max_ew_tokes=100)

result = tokeizer.decode(outputs[0], skip_special_tokes=True)
prit(result)

Chat Completio

import torch
from modelscope import AutoTokeizer, AutoModelForCausalLM, GeeratioCofig

model_ame = "deepseek-ai/DeepSeek-V2-Chat"
tokeizer = AutoTokeizer.from_pretraied(model_ame, trust_remote_code=True)
# `max_memory` should be set based o your devices
max_memory = {i: "75GB" for i i rage(8)}
model = AutoModelForCausalLM.from_pretraied(model_ame, trust_remote_code=True, device_map="auto", torch_dtype=torch.bfloat16, max_memory=max_memory)
model.geeratio_cofig = GeeratioCofig.from_pretraied(model_ame)
model.geeratio_cofig.pad_toke_id = model.geeratio_cofig.eos_toke_id

messages = [
    {"role": "user", "cotet": "Write a piece of quicksort code i C++"}
]
iput_tesor = tokeizer.apply_chat_template(messages, add_geeratio_prompt=True, retur_tesors="pt")
outputs = model.geerate(iput_tesor.to(model.device), max_ew_tokes=100)

result = tokeizer.decode(outputs[0][iput_tesor.shape[1]:], skip_special_tokes=True)
prit(result)

The complete chat template ca be foud withi tokeizer_cofig.jso located i the huggigface model repository.

A example of chat template is as belows:

<｜begi▁of▁setece｜>User: {user_message_1}

Assistat: {assistat_message_1}<｜ed▁of▁setece｜>User: {user_message_2}

Assistat:

You ca also add a optioal system message:

<｜begi▁of▁setece｜>{system_message}

User: {user_message_1}

Assistat: {assistat_message_1}<｜ed▁of▁setece｜>User: {user_message_2}

Assistat:

8. Licese

This code repository is licesed uder the MIT Licese. The use of DeepSeek-V2 Base/Chat models is subject to the Model Licese. DeepSeek-V2 series (icludig Base ad Chat) supports commercial use.

9. Citatio

@misc{deepseek-v2,
  author = {DeepSeek-AI},
  title  = {DeepSeek-V2: A Strog, Ecoomical, ad Efficiet Mixture-of-Experts Laguage Model},
  year   = {2024},
  ote   = {GitHub repository},
  url    = {https://github.com/deepseek-ai/deepseek-v2}
  }

10. Cotact

If you have ay questios, please raise a issue or cotact us at service@deepseek.com.

Model Download | Evaluation Results | Model Architecture | API Platform | License | Citati