• What is Yi?
• Introduction
• Models
  • Chat models
  • Base models
  • Model info
• News
• How to use Yi?
• Quick start
  • Choose your path
  • pip
  • docker
  • llama.cpp
  • conda-lock
  • Web demo
• Fine-tuning
• Quantization
• Deployment
• FAQ
• Learning hub
• Why Yi?
• Ecosystem
  • Upstream
  • Downstream
  • Serving
  • Quantization
  • Fine-tuning
  • API
• Benchmarks
  • Base model performance
  • Chat model performance
• Tech report
  • Citation
• Who can use Yi?
• Misc.
• Acknowledgements
• Disclaimer
• License

• Yi-34B-Chat
• Yi-34B-Chat-4bits
• Yi-34B-Chat-8bits
• Yi-6B-Chat
• Yi-6B-Chat-4bits
• Yi-6B-Chat-8bits

You can try some of them interactively at:

• Hugging Face
• Replicate

• English
• Chinese

Prompt: Let me tell you an interesting story about cat Tom and mouse Jerry,

Generation: Let me tell you an interesting story about cat Tom and mouse Jerry, which happened in my childhood. My father had a big house with two cats living inside it to kill mice. One day when I was playing at home alone, I found one of the tomcats lying on his back near our kitchen door, looking very much like he wanted something from us but couldn’t get up because there were too many people around him! He kept trying for several minutes before finally giving up…

Step 0: Prerequisites

Make sure you've installed Docker and nvidia-container-toolkit.

Step 1: Start Docker

docker run -it --gpus all \
-v <your-model-path>: /models
ghcr.io/01-ai/yi:latest

Alternatively, you can pull the Yi Docker image from registry.lingyiwanwu.com/ci/01-ai/yi:latest.

Step 2: Perform inference

<p>You can perform inference with Yi chat or base models as below.</p>

Perform inference with Yi chat model

<p>The steps are similar to <a href="#perform-inference-with-yi-chat-model">pip - Perform inference with Yi chat model</a>.</p>
<p><strong>Note</strong> that the only difference is to set model_path = '&lt;your-model-mount-path&gt;' instead of model_path = '&lt;your-model-path&gt;'.</p>

Perform inference with Yi base model

<p>The steps are similar to <a href="#perform-inference-with-yi-base-model">pip - Perform inference with Yi base model</a>.</p>
<p><strong>Note</strong> that the only difference is to set --model &lt;your-model-mount-path&gt;' instead of model &lt;your-model-path&gt;.</p>

1. Install micromamba by following the instructions available here.

2. Execute micromamba install -y -n yi -f conda-lock.yml to create a conda environment named yi and install the necessary dependencies.

• Step 0: Prerequisites
• Step 1: Download llama.cpp
• Step 2: Download Yi model
• Step 3: Perform inference

Step 0: Prerequisites

• This tutorial assumes you use a MacBook Pro with 16GB of memory and an Apple M2 Pro chip.

• Make sure git-lfs is installed on your machine.

Step 1: Download llama.cpp

To clone the llama.cpp repository, run the following command.

git clone git@github.com:ggerganov/llama.cpp.git

Step 2: Download Yi model

2.1 To clone XeIaso/yi-chat-6B-GGUF with just pointers, run the following command.

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/XeIaso/yi-chat-6B-GGUF

2.2 To download a quantized Yi model (yi-chat-6b.Q2_K.gguf), run the following command.

git-lfs pull --include yi-chat-6b.Q2_K.gguf

Step 3: Perform inference

To perform inference with the Yi model, you can use one of the following methods.

• Method 1: Perform inference in terminal

• Method 2: Perform inference in web

Method 1: Perform inference in terminal

To compile llama.cpp using 4 threads and then conduct inference, navigate to the llama.cpp directory, and run the following command.

Tips

• Replace /Users/yu/yi-chat-6B-GGUF/yi-chat-6b.Q2_K.gguf with the actual path of your model.

• By default, the model operates in completion mode.

• For additional output customization options (for example, system prompt, temperature, repetition penalty, etc.), run ./main -h to check detailed descriptions and usage.

make -j4 && ./main -m /Users/yu/yi-chat-6B-GGUF/yi-chat-6b.Q2_K.gguf -p "How do you feed your pet fox? Please answer this question in 6 simple steps:\nStep 1:" -n 384 -e

...

How do you feed your pet fox? Please answer this question in 6 simple steps:

Step 1: Select the appropriate food for your pet fox. You should choose high-quality, balanced prey items that are suitable for their unique dietary needs. These could include live or frozen mice, rats, pigeons, or other small mammals, as well as fresh fruits and vegetables.

Step 2: Feed your pet fox once or twice a day, depending on the species and its individual preferences. Always ensure that they have access to fresh water throughout the day.

Step 3: Provide an appropriate environment for your pet fox. Ensure it has a comfortable place to rest, plenty of space to move around, and opportunities to play and exercise.

Step 4: Socialize your pet with other animals if possible. Interactions with other creatures can help them develop social skills and prevent boredom or stress.

Step 5: Regularly check for signs of illness or discomfort in your fox. Be prepared to provide veterinary care as needed, especially for common issues such as parasites, dental health problems, or infections.

Step 6: Educate yourself about the needs of your pet fox and be aware of any potential risks or concerns that could affect their well-being. Regularly consult with a veterinarian to ensure you are providing the best care.

...

Now you have successfully asked a question to the Yi model and got an answer! ?

Method 2: Perform inference in web

1. To initialize a lightweight and swift chatbot, run the following command.

   cd llama.cpp
   ./server --ctx-size 2048 --host 0.0.0.0 --n-gpu-layers 64 --model /Users/yu/yi-chat-6B-GGUF/yi-chat-6b.Q2_K.gguf
   

   Then you can get an output like this:

   ...
   
   llama_new_context_with_model: n_ctx      = 2048
   llama_new_context_with_model: freq_base  = 5000000.0
   llama_new_context_with_model: freq_scale = 1
   ggml_metal_init: allocating
   ggml_metal_init: found device: Apple M2 Pro
   ggml_metal_init: picking default device: Apple M2 Pro
   ggml_metal_init: ggml.metallib not found, loading from source
   ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
   ggml_metal_init: loading '/Users/yu/llama.cpp/ggml-metal.metal'
   ggml_metal_init: GPU name:   Apple M2 Pro
   ggml_metal_init: GPU family: MTLGPUFamilyApple8 (1008)
   ggml_metal_init: hasUnifiedMemory              = true
   ggml_metal_init: recommendedMaxWorkingSetSize  = 11453.25 MB
   ggml_metal_init: maxTransferRate               = built-in GPU
   ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =   128.00 MiB, ( 2629.44 / 10922.67)
   llama_new_context_with_model: KV self size  =  128.00 MiB, K (f16):   64.00 MiB, V (f16):   64.00 MiB
   ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     0.02 MiB, ( 2629.45 / 10922.67)
   llama_build_graph: non-view tensors processed: 676/676
   llama_new_context_with_model: compute buffer total size = 159.19 MiB
   ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =   156.02 MiB, ( 2785.45 / 10922.67)
   Available slots:
   -> Slot 0 - max context: 2048
   
   llama server listening at http://0.0.0.0:8080
   

2. To access the chatbot interface, open your web browser and enter http://0.0.0.0:8080 into the address bar.

   

3. Enter a question, such as "How do you feed your pet fox? Please answer this question in 6 simple steps" into the prompt window, and you will receive a corresponding answer.

   

Finetune code for Yi 6B and 34B

Preparation

From Image

By default, we use a small dataset from BAAI/COIG to finetune the base model. You can also prepare your customized dataset in the following jsonl format:

{ "prompt": "Human: Who are you? Assistant:", "chosen": "I'm Yi." }

And then mount them in the container to replace the default ones:

docker run -it \
    -v /path/to/save/finetuned/model/:/finetuned-model \
    -v /path/to/train.jsonl:/yi/finetune/data/train.json \
    -v /path/to/eval.jsonl:/yi/finetune/data/eval.json \
    ghcr.io/01-ai/yi:latest \
    bash finetune/scripts/run_sft_Yi_6b.sh

From Local Server

Make sure you have conda. If not, use

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
~/miniconda3/bin/conda init bash
source ~/.bashrc

Then, create a conda env:

conda create -n dev_env python=3.10 -y
conda activate dev_env
pip install torch==2.0.1 deepspeed==0.10 tensorboard transformers datasets sentencepiece accelerate ray==2.7

Hardware Setup

For the Yi-6B model, a node with 4 GPUs, each with GPU memory larger than 60GB, is recommended.

For the Yi-34B model, because the usage of the zero-offload technique consumes a lot of CPU memory, please be careful to limit the number of GPUs in the 34B finetune training. Please use CUDAVISIBLEDEVICES to limit the number of GPUs (as shown in scripts/runsftYi_34b.sh).

A typical hardware setup for finetuning the 34B model is a node with 8 GPUs (limited to 4 in running by CUDAVISIBLEDEVICES=0,1,2,3), each with GPU memory larger than 80GB, and total CPU memory larger than 900GB.

Quick Start

Download a LLM-base model to MODEL_PATH (6B and 34B). A typical folder of models is like:

|-- $MODEL_PATH
|   |-- config.json
|   |-- pytorch_model-00001-of-00002.bin
|   |-- pytorch_model-00002-of-00002.bin
|   |-- pytorch_model.bin.index.json
|   |-- tokenizer_config.json
|   |-- tokenizer.model
|   |-- ...

Download a dataset from huggingface to local storage DATA_PATH, e.g. Dahoas/rm-static.

|-- $DATA_PATH
|   |-- data
|   |   |-- train-00000-of-00001-2a1df75c6bce91ab.parquet
|   |   |-- test-00000-of-00001-8c7c51afc6d45980.parquet
|   |-- dataset_infos.json
|   |-- README.md

finetune/yi_example_dataset has example datasets, which are modified from BAAI/COIG

|-- $DATA_PATH
    |--data
        |-- train.jsonl
        |-- eval.jsonl

cd into the scripts folder, copy and paste the script, and run. For example:

cd finetune/scripts

bash run_sft_Yi_6b.sh

For the Yi-6B base model, setting trainingdebugsteps=20 and numtrainepochs=4 can output a chat model, which takes about 20 minutes.

For the Yi-34B base model, it takes a relatively long time for initialization. Please be patient.

Evaluation

cd finetune/scripts

bash run_eval.sh

Then you'll see the answer from both the base model and the finetuned model.

GPT-Q quantization

GPT-Q is a PTQ (Post-Training Quantization) method. It saves memory and provides potential speedups while retaining the accuracy of the model.

Yi models can be GPT-Q quantized without a lot of efforts. We provide a step-by-step tutorial below.

To run GPT-Q, we will use AutoGPTQ and exllama. And the huggingface transformers has integrated optimum and auto-gptq to perform GPTQ quantization on language models.

Do Quantization

The quant_autogptq.py script is provided for you to perform GPT-Q quantization:

python quant_autogptq.py --model /base_model \
    --output_dir /quantized_model --bits 4 --group_size 128 --trust_remote_code

Run Quantized Model

You can run a quantized model using the eval_quantized_model.py:

python eval_quantized_model.py --model /quantized_model --trust_remote_code

AWQ quantization

AWQ is a PTQ (Post-Training Quantization) method. It's an efficient and accurate low-bit weight quantization (INT3/4) for LLMs.

Yi models can be AWQ quantized without a lot of efforts. We provide a step-by-step tutorial below.

To run AWQ, we will use AutoAWQ.

Do Quantization

The quant_autoawq.py script is provided for you to perform AWQ quantization:

python quant_autoawq.py --model /base_model \
    --output_dir /quantized_model --bits 4 --group_size 128 --trust_remote_code

Run Quantized Model

You can run a quantized model using the eval_quantized_model.py:

python eval_quantized_model.py --model /quantized_model --trust_remote_code

?Fine-tuning

• Base model or Chat model - which to fine-tune?
  The choice of pre-trained language model for fine-tuning hinges on the computational resources you have at your disposal and the particular demands of your task.
  • If you are working with a substantial volume of fine-tuning data (say, over 10,000 samples), the Base model could be your go-to choice.
  • On the other hand, if your fine-tuning data is not quite as extensive, opting for the Chat model might be a more fitting choice.
  • It is generally advisable to fine-tune both the Base and Chat models, compare their performance, and then pick the model that best aligns with your specific requirements.
• Yi-34B versus Yi-34B-Chat for full-scale fine-tuning - what is the difference?
  The key distinction between full-scale fine-tuning on Yi-34Band Yi-34B-Chat comes down to the fine-tuning approach and outcomes.
  • Yi-34B-Chat employs a Special Fine-Tuning (SFT) method, resulting in responses that mirror human conversation style more closely.
  • The Base model's fine-tuning is more versatile, with a relatively high performance potential.
  • If you are confident in the quality of your data, fine-tuning with Yi-34B could be your go-to.
  • If you are aiming for model-generated responses that better mimic human conversational style, or if you have doubts about your data quality, Yi-34B-Chat might be your best bet.

?Quantization

• Quantized model versus original model - what is the performance gap?
  • The performance variance is largely contingent on the quantization method employed and the specific use cases of these models. For instance, when it comes to models provided by the AWQ official, from a Benchmark standpoint, quantization might result in a minor performance drop of a few percentage points.
  • Subjectively speaking, in situations like logical reasoning, even a 1% performance shift could impact the accuracy of the output results.

?General

• Where can I source fine-tuning question answering datasets?

  • You can find fine-tuning question answering datasets on platforms like Hugging Face, with datasets like m-a-p/COIG-CQIA readily available.
  • Additionally, Github offers fine-tuning frameworks, such as hiyouga/LLaMA-Factory, which integrates pre-made datasets.

• What is the GPU memory requirement for fine-tuning Yi-34B FP16?
  The GPU memory needed for fine-tuning 34B FP16 hinges on the specific fine-tuning method employed. For full parameter fine-tuning, you'll need 8 GPUs each with 80 GB; however, more economical solutions like Lora require less. For more details, check out hiyouga/LLaMA-Factory. Also, consider using BF16 instead of FP16 for fine-tuning to optimize performance.

• Are there any third-party platforms that support chat functionality for the Yi-34b-200k model?
  If you're looking for third-party Chats, options include fireworks.ai.

Welcome to the Yi learning hub!

Whether you're a seasoned developer or a newcomer, you can find a wealth of helpful educational resources to enhance your understanding and skills with Yi models, including insightful blog posts, comprehensive video tutorials, hands-on guides, and more.

The content you find here has been generously contributed by knowledgeable Yi experts and passionate enthusiasts. We extend our heartfelt gratitude for your invaluable contributions!

At the same time, we also warmly invite you to join our collaborative effort by contributing to Yi. If you have already made contributions to Yi, please don't hesitate to showcase your remarkable work in the table below.

With all these resources at your fingertips, you're ready to start your exciting journey with Yi. Happy learning! ?

Tutorials

Blog tutorials

GitHub Project

Video tutorials