ChatGLM Efficient Tuning

Fine-tuning 🤖ChatGLM-6B model with 🤗PEFT.

Datasets

Our script now supports the following datasets:

• Stanford Alpaca
• Stanford Alpaca (Chinese)
• GPT-4 Generated Data
• BELLE 2M
• BELLE 1M
• BELLE 0.5M
• BELLE Dialogue 0.4M
• BELLE School Math 0.25M
• BELLE Multiturn Chat 0.8M
• Guanaco Dataset
• Firefly 1.1M

Please refer to config_data.py for details.

Fine-Tuning Methods

Our script now supports the following fine-tuning methods:

• P-Tuning V2
• LoRA

Requirement

• Python 3.10 and PyTorch 2.0.0
• 🤗Transformers, Datasets, and PEFT
• protobuf, cpm_kernels, sentencepiece

And powerful GPUs!

Getting Started

Preparation

git clone https://github.com/hiyouga/ChatGLM-Efficient-Tuning.git
conda create -n cet python=3.10
conda activate cet
pip install -r requirements.txt

Fine-tuning

CUDA_VISIBLE_DEVICES=0 python finetune_chatglm.py \
    --do_train \
    --dataset guanaco \
    --output_dir output_guanaco \
    --overwrite_cache \
    --overwrite_output_dir \
    --per_device_train_batch_size 4 \
    --gradient_accumulation_steps 4 \
    --lr_scheduler_type cosine \
    --logging_steps 10 \
    --save_steps 1000 \
    --warmup_steps 100 \
    --max_train_samples 10000 \
    --learning_rate 5e-4 \
    --num_train_epochs 1.0 \
    --fp16

Inference

CUDA_VISIBLE_DEVICES=0 python infer_chatglm.py

Hardware Requirements

Batch size	LoRA r	Mode	GRAM
8	8	FP16	24GB

Compared with Existing Implementations

• THUDM/ChatGLM-6B
  • Official implementation of fine-tuning ChatGLM with P-Tuning v2 on the ADGEN dataset.
  • Our fine-tuning script is largely depend on it. We further implement the LoRA tuning method. Additionally, we dynamically pad the inputs to the longest sequence in the batch instead of the maximum length.
• mymusise/ChatGLM-Tuning
  • An unoffical implementation of fine-tuning ChatGLM with LoRA on the Stanford Alpaca dataset.
  • We borrowed some ideas from it. Our fine-tuning script integrates the data pre-processing part into the training procedure, so we need not generate a pre-processed dataset before training.
• ssbuild/chatglm_finetuning
  • An unofficial implementation of fine-tuning ChatGLM with several PEFT methods on the Stanford Alpaca dataset.
  • Our fine-tuning script is implemented purely with Huggingface transformers and is independent of the deep_training framework.
• lich99/ChatGLM-finetune-LoRA
  • An unofficial implementation of fine-tuning ChatGLM with LoRA on the Stanford Alpaca dataset.
  • We use the Huggingface PEFT to provide the state-of-the-art PEFT methods.
• liucongg/ChatGLM-Finetuning
  • An unofficial implementation of fine-tuning ChatGLM with several methods including Freeze, LoRA and P-Tuning on the industrial dataset.
  • We are aim to incorporate more instruction-following datasets for fine-tuning the ChatGLM model.
• yanqiangmiffy/InstructGLM
  • An unofficial implementation of fine-tuning ChatGLM that explores the ChatGLM's ability on the instruction-following datasets.
  • Our fine-tuning script integrates the data pre-processing part in to the training procedure.

TODO

• Incorporating Chinese datasets into the training sets.
  • BELLE
  • pCLUE
  • CLUECorpus
  • GuanacoDataset
  • FireflyDataset
• Incorporating ChatGPT & GPT-4 self-chat data into the training sets.
  • Baize
  • GPT-4-LLM
• Implementing the Freeze-Tuning and P-Tuning method.
• Supporting Multi-GPUs fine-tuning.

License

This repository is licensed under the Apache-2.0 License.

Citation

If this work is helpful, please cite as:

@Misc{chatglm-efficient-tuning,
  title = {ChatGLM Efficient Tuning},
  author = {hiyouga},
  howpublished = {\url{https://github.com/hiyouga/ChatGLM-Efficient-Tuning}},
  year = {2023}
}

Acknowledgement

This repo benefits from ChatGLM-6B, ChatGLM-Tuning and yuanzhoulvpi2017/zero_nlp. Thanks for their wonderful works.