Add local Llama2 support from llama2-wrapper backend

[liltom-eth]

Hi @Shaunwei @pycui ,
I am working on the project llama2-wrapper to make it easily call Llama2 model locally as an LLM backend.
And to follow up on the Twitter discussion, I made this PR as a showcase running Realchar and Llama2 locally on an M2 Macbook Air.
Here is the demo:

How to run on Mac:

Run OpenAI Compatible API on Llama2 models

pip install llama2-wrapper
python -m llama2_wrapper.server  --port 8001
# Llama2 running on http://localhost:8001

Start Realchar

python cli.py web-build
python cli.py run-uvicorn

Implementation

I found it hard to load local LLM object directly as backend since Realchar is using langchain.chat_models as the LLM.
Thus I chose to run a local LLM as OpenAI Compatible API, then call langchain.chat_models.ChatOpenAI to run LLM from the local URL.

Issues

Now the PR still has issues automatically passing customize URL from .env as the model URL to llm. I haven't figured out how to add a new LLM option in the new Realchar Web UI and hard code to make Realchar run on llama2-wrapper.

Showcase

This showcase is running Realchar and Llama2 on Mac. (13.70 tokens/sec through llama.cpp)
Another interesting showcase might be running Realchar and Llama2 on free colab T4 GPU. (18.19 tokens/sec through gptq)

[pycui]

Thanks for making this! A few comments

[pycui]

can make this an env

[pycui]

let's keep the branching logic for the formal PR

[pycui]

also might need a convention to route to local, e.g. maybe just call it local for now

[liltom-eth]

Thanks! I found that OPENAI_API_KEY in .env is always required and if not will raise the error:

openai.error.AuthenticationError: Incorrect API key provided: YOUR_API_KEY. You can find your API key at https://platform.openai.com/account/api-keys.

[liltom-eth]

let's keep the branching logic for the formal PR

If I keep the branching logic here, the arg model will always be "gpt-3.5-turbo-16k", then initial a OpenaiLlm.
I think the reason is that there is no local button on frontend, and my choice GPT-3.5 will always set arg model as gpt-3.5-turbo-16k.
And LLM_MODEL_USE from .env is overwritten by frontend choice.

[johndpope]

(presumably this is unusable on 3090 / too slow, right? ) @liltom-eth - do you have an a100 - or 2x 4090s?

[pycui]

I understand your current code makes showcasing the demo easier, but for us to merge the code base we should still aim to incorporate with existing logic. I suggest we first make the backend part ready.

For the frontend selection, we can make an environment variable or UI advanced option to enable local Llama inference. When this is toggled, the model string passed to the backend can be your choice here in the backend. The frontend part can be a separate PR if you would like. For testing only, you can change the model string of the "Llama-2-70b" to test.

[liltom-eth]

(presumably this is unusable on 3090 / too slow, right? ) @liltom-eth - do you have an a100 - or 2x 4090s?

I believe it is usable on 3090, (running gptq model 18.85 tokens/sec on 2080ti).
But right now when I was running on Windows WSL2 to demo on 2080ti, I got some errors on Realchar.

[liltom-eth]

I understand your current code makes showcasing the demo easier, but for us to merge the code base we should still aim to incorporate with existing logic. I suggest we first make the backend part ready.

For the frontend selection, we can make an environment variable or UI advanced option to enable local Llama inference. When this is toggled, the model string passed to the backend can be your choice here in the backend. The frontend part can be a separate PR if you would like. For testing only, you can change the model string of the "Llama-2-70b" to test.

Thank you! I will test it by using "Llama-2-70b" button in this PR. Another PR for frontend would be helphul.

[liltom-eth]

@pycui When I tried the frontend button "Llama-2-70b", it always through an error like:

Is that error happening because of checking anyscale key?

[pycui]

It seems because using a non-3.5 model directs you to a firebase auth, but you probably don't have a working Firebase App. You can probably edit thisclient/web/src/App.jsx L218, so that (in your test) your model name doesn't require a sign-in.

[johndpope]

before this gets merged - theres some caveats to be mindful with these local llms.
mostly - the model supplied by facebook out of the box is somewhat unusable on consumer hardware. I'm not sure if this pr is directly targeting that file format -
it has high floating point precision making model huge and vram intensive.
here's an article explaining ins and out of this.
https://brandolosaria.medium.com/setting-up-metaais-code-llama-34b-instruct-model-fc009aa937f6

so everyone is using the quantized / smaller 4 or 5bit models to get anything usable.
and they also use hugging face to download the models. so it becomes trivial to get the latest models / using text-generation-webui.
eg. there's been 2 new ones for codellama in last 24hrs.

there's also contention on what models get merged - and this becomes tech spike
I raised this issue in another repo suggesting to yield to a flexible upstream model provider
nomic-ai/gpt4all#1238

this one seems great - then it's their problem to update the models.
https://github.com/lmstudio-ai/model-catalog/blob/main/catalog.json

[liltom-eth]

@pycui Thank you! Have made some updates based on your suggestions.
If I select "Llama-2-70b" button from frontend, model here will be "meta-llama/Llama-2-70b-chat-hf".
Thus I load model temporarily from .env here.

[pycui]

Thanks, I made some change to still use the model param. For testing, you can modify the frontend to pass localhost as model name.

[pycui]

Suggested change

	# LOCAL_LLM_URL="http://localhost:8001/v1"
	# Example value: "http://localhost:8001/v1"

[liltom-eth]

OK.

[pycui]

remove this?

[liltom-eth]

Thanks! Made an update to clean this.

[liltom-eth]

before this gets merged - theres some caveats to be mindful with these local llms. mostly - the model supplied by facebook out of the box is somewhat unusable on consumer hardware. I'm not sure if this pr is directly targeting that file format - it has high floating point precision making model huge and vram intensive. here's an article explaining ins and out of this. https://brandolosaria.medium.com/setting-up-metaais-code-llama-34b-instruct-model-fc009aa937f6

so everyone is using the quantized / smaller 4 or 5bit models to get anything usable. and they also use hugging face to download the models. so it becomes trivial to get the latest models / using text-generation-webui. eg. there's been 2 new ones for codellama in last 24hrs.

there's also contention on what models get merged - and this becomes tech spike I raised this issue in another repo suggesting to yield to a flexible upstream model provider nomic-ai/gpt4all#1238

this one seems great - then it's their problem to update the models. https://github.com/lmstudio-ai/model-catalog/blob/main/catalog.json

Thanks! That is a good idea. A model catalog can be helpful for users and developers.