Add Baichuan2 Support

[AoyuQC]

Add support for baichuan2, issue #50

[casper-hansen]

This looks great, thanks for the PR @AoyuQC. Could you run perplexity on FP16 vs INT4 quantized so we can see how much performance is degraded?

python examples/eval.py --model_path <your_model>

[AoyuQC]

Hi, @casper-hansen , I have run perplexity test on FP16 vs INT4. The FP16 version is 6.800 while the INT4 version is 6.938. Please check the following image for experiment results.

[casper-hansen]

Solid numbers @AoyuQC! And great work :)

I have a few questions:

1. You have created a BaichuanBlock. It looks a lot like the LlamaLikeBlock. Can we reuse the existing one? (we already reuse it for mistral, yi, qwen, etc.)
2. Is the modification to calib_data.py necessary to support the model? It seems like the new method makes some new assumptions about the format of custom datasets.

[AoyuQC]

Solid numbers @AoyuQC! And great work :)

I have a few questions:

1. You have created a `BaichuanBlock`. It looks a lot like the `LlamaLikeBlock`. Can we reuse the existing one? (we already reuse it for mistral, yi, qwen, etc.)

2. Is the modification to `calib_data.py` necessary to support the model? It seems like the new method makes some new assumptions about the format of custom datasets.

Hi @casper-hansen , I have reused LlamaLikeBlock for baichuan2, please have a look. The reason that I modified calib_data.py is that I want to support the case that people want to use tokenized input for calibration not just raw string input.