Llama 3.1 - not working with home-llm

[WW1983]

llama 3.1 is out. Currently, however, it does not work with Home Assistant. Or did someone make it work?

[franco-giordano]

I'm currently using it fine, having upgrade it from Llama3. What exactly is not working in your case? I'm using Ollama as the backend, and Llama3.1-8b

[kltye]

To kill several birds with one stone (or answer the multiple helpful replies):

1. I'm using LLM Model 'llama3.1:8b-instruct-q8_0'

2. I'm using the default prompt for Llama models given by the HACS integration

3. The "Use chat completions endpoint" toggle is set to OFF

4. I'm running HA 2024.7.4.

5. I'm running Ollama on Windows by just pulling the llama3.1 model above and doing "ollama serve". Everything else is default (except for setting the OLLAMA_HOST environment variable to my local IP address).

6. I've restarted HA and there's been no difference :(

Hey, just to check: are you exposing the devices to the Assistant? You can check that in Voice Assistants > Expose (https://my.home-assistant.io/redirect/voice_assistants/) Otherwise no devices will be shown in the prompt. Also, I'd enable debug logging for the integration (Integrations > Local LLM Conversation > Enable Debug logging) and see in the logs what's the prompt and response that the LLM uses/generates.

edit: also, Prompt Format: Llama 3, JSON output mode: OFF

Yup, using Ollama and the mistral 7b instruct model works just fine, and debug logging does show that my entities and prompt are being sent over to the model. And yes, the Llama 3 prompt format is selected with JSON output set to OFF. FWIW, I'm also having the same "model wants to chat instead of send commands" problem with the Phi-3 instruct model as well (using the Zephyr prompt format), though I'm not sure if that's ever been confirmed to work with this.

[muhlba91]

on my side, manually adapting the examples fixes the issue that the model tries to select the wrong entity for actions and it also fixes the usage of entity id's and friendly names.

however, now i got an interesting one instead. i have 3 aircons exposing climate entities and in each room i also have temperature sensors which are part of motion sensors. their friendly name is pretty similar (e.g. "living room air conditioner", "living room motion sensor - temperature"). when querying for "what is the living room temperature?" the model always selects the climate entity and i'm not able to find a prompt that changes this. in fact, if i ask for the temperature in a room without a climate entity but a motion sensor, the model still replies that there's no temperature for this room and there are only rooms xyz with a climate entity.

any ideas on how to tune the prompt here?

[franco-giordano]

@kltye that's really weird, sadly I'm out of ideas :( The last thing I would try is to manually talk to the model via ollama run ... and paste the same prompt that is sent to the model, just to check whether it is home-llm's fault or not. Also try updating the model (ollama pull ...) in case it is using an old version. Other than that, no clue 🤔

@muhlba91 nice! Personally I don't have a setup like so, but I would try adding more examples showing different uses for that (where you use the motion sensors for temp retrieval). You could also add examples where no tool is needed at all. For example, I have a weather entity and so I added an example pair like "What's the weather like / The weather is ..." without a <functioncall> block. This way I tried showing the model that it is OK to skip function call. Another option would be to change or add a new friendly name to the sensors, so that they are more "obvious" (i.e. remove the motion sensor part). Last option would be to simply tell the model in the prompt (maybe you already tried this) something like "If the user asks about the temperature in the room, don't use the air conditioners and instead use the motion sensors data to respond"

[kltye]

@kltye that's really weird, sadly I'm out of ideas :( The last thing I would try is to manually talk to the model via ollama run ... and paste the same prompt that is sent to the model, just to check whether it is home-llm's fault or not. Also try updating the model (ollama pull ...) in case it is using an old version. Other than that, no clue 🤔

I just did manually talk to the model (using the debug output from Home LLM) but all of them (including mistral) is somewhat confused and just wants to talk to me as well. I'm not familiar with model prompting and programming, but I'm pretty sure I included the various tokens (?) that indicate the end of instructions, etc. I suppose there are other parameters that might be sent along with an API request that isn't reflected when simply chatting with the model.

[kltye]

UPDATE: It looks like the default context length of 2048 was too short. I increased it to 4000 and it seems to be mostly working correctly now (at least it tries to execute commands rather than chat with me). There are some errors with slightly more complicated device names likes groups, but I'll tweak the settings further. Thanks for everyone's help!