-
Notifications
You must be signed in to change notification settings - Fork 9.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug: Model ignores system prompt when use /completion
endpoint
#8393
Comments
@andreys42 Unless you are using conversation llama-cli -cnv mode you will need to use --in-prefix --in-suffix or wrap your input in Llama3 prompt template. |
@dspasyuk tnx for suggestion, |
You are probably using the wrong template. Send your request to the You can compare the good and bad prompts to see what was wrong. |
@andreys42 here is the setting I use in llama.cui that works well across major models: ../llama.cpp/llama-cli --model ../../models/meta-llama-3-8b-instruct-q5_k_s.gguf --n-gpu-layers 25 -cnv --simple-io -b 2048 --ctx_size 0 --temp 0 --top_k 10 --multiline-input --chat-template llama3 --log-disable Here is the result: Screencast.from.2024-07-10.10.20.44.AM.webmYou can test it for yourself here: https://github.com/dspasyuk/llama.cui |
This issue was closed because it has been inactive for 14 days since being marked as stale. |
What happened?
I'm testing the Meta-Llama-3-8B-Instruct-Q8_0 model using the llamacpp HTTP server, both through the
chatui
interface and direct requests via Python's requests.When I use
chatui
with thechatPromptTemplate
option, everything works fine, and the model's output is predictable and desirable.However, when I make direct requests to the same server with the same model, the output is messy (lot of
newline
characters, repeating of the question, and so on) and most of the system instructions are being ignored (but general logic of ouput is fine), when I ask to answer only with 0 or 1, model still trying to motivate its decision in outputMy attempts so far have been:
I've spent a lot of time trying to figure out the issue, but all of these approaches work much worse than using
chatui
way.I believe the problem lies in my understanding of how to format the input prompts, and I'm not familiar enough with the syntax documentation.
Name and Version
lastest libs
Meta-Llama-3-8B-Instruct-Q8_0
What operating system are you seeing the problem on?
No response
Relevant log output
No response
The text was updated successfully, but these errors were encountered: