Model Card for CodeFuse-13B

[中文] [Eglish]

Model Descriptio

CodeFuse-13B is a 13 billio parameter code geeratio model traied o the GPT-NeoX framework, capable of hadlig code sequeces of up to 4096 characters. This model was pretraied o a dataset cosistig of 1000B toke code, Chiese, ad Eglish data, coverig over 40 programmig laguages. To further ehace the effectiveess ad quality of the geerated code, the model was fie-tued o the CodeFuse-Evol-istructio-66k dataset, eablig it to produce more accurate, efficiet, ad compliat code. Pass@1 achieved 37.1% o the HumaEval evaluatio set(BeamSearch strategy, BeamSize=3).

Code Commuity

Homepage: ? https://github.com/codefuse-ai (Please give us your support with a Star? + Fork? + Watch?)

• If you wish to fie-tue the model yourself, you ca visit ✨MFTCoder✨✨

• If you wish to deploy the model yourself, you ca visit ✨FasterTrasformer4CodeFuse✨✨

• If you wish to see a demo of the model, you ca visit ✨CodeFuse Demo✨✨

Requiremets

• Pytho 3.8 or above.
• PyTorch 1.12 or above, with a recommedatio for 2.0 or above.
• Trasformers 4.24.0 or above.
• It is advisable to use CUDA 11.4 or above (for GPU users ad flash-attetio users, this optio should be cosidered).

Quickstart

import torch
from modelscope import AutoModelForCausalLM, AutoTokeizer, sapshot_dowload

model_dir = sapshot_dowload('codefuse-ai/CodeFuse-13B', revisio='v1.0.0')
tokeizer = AutoTokeizer.from_pretraied(model_dir)
model = AutoModelForCausalLM.from_pretraied(model_dir, device_map="auto", torch_dtype=torch.float16).eval()

iput_ids = tokeizer.ecode("# laguage: Pytho\def quick_sort(array):\", retur_tesors="pt").to("cuda")
output_ids = model.geerate(iput_ids, max_ew_tokes=200)

prit(tokeizer.decode(output_ids[0]))

MD5

We otice that the file may be corrupted durig trasfer process. Please check MD5 value before use.

简介

CodeFuse-13B是基于GPT-NeoX框架训练的13B参数代码生成模型，能够处理4096个字符的代码序列。该模型在1000B Toke的代码、中文、英文数据数据集上进行预训练，覆盖超过40种编程语言。为了进一步提升生成代码的效果和质量，该模型还在CodeFuse-Evol-istructio-66k数据集上进行了微调，使得该模型能够生成更加准确、高效、符合要求的代码。在HumaEval评测集上Pass@1达到37.1%(采用BeamSearch解码，其中BeamSize=3)。

代码社区

大本营： ? https://github.com/codefuse-ai （请支持我们的项目 Star?&bsp;+ Fork? + Watch?）

• 如果您想自己微调该模型，可以访问 ✨MFTCoder✨✨

• 如果您想自己部署该模型，可以访问 ✨FasterTrasformer4CodeFuse✨✨

• 如果您想观看该模型示例，可以访问 ✨CodeFuse Demo✨✨

要求

• pytho 3.8及以上版本
• pytorch 1.12及以上版本，推荐2.0及以上版本
• trasformers 4.24.0及以上版本
• 建议使用CUDA 11.4及以上（GPU用户、flash-attetio用户等需考虑此选

快速使用

import torch
from modelscope import AutoModelForCausalLM, AutoTokeizer, sapshot_dowload

model_dir = sapshot_dowload('codefuse-ai/CodeFuse-13B', revisio='v1.0.0')
tokeizer = AutoTokeizer.from_pretraied(model_dir)
model = AutoModelForCausalLM.from_pretraied(model_dir, device_map="auto", torch_dtype=torch.float16).eval()

iput_ids = tokeizer.ecode("# laguage: Pytho\def quick_sort(array):\", retur_tesors="pt").to("cuda")
output_ids = model.geerate(iput_ids, max_ew_tokes=200)

prit(tokeizer.decode(output_ids[0]))

MD5

我们发现模型文件可能会在传输过程中损坏，使用前请检查文件MD5值。

Model Card for CodeFuse-13B [中文] [English] Model Description CodeFuse-13B is a 13 billion