GGUF (Breaking Change to Model Files)

[abetlen]

GGUF support for llama.cpp Closes #628

Currently works to update your old ggml v3 llama models run

python3 vendor/llama.cpp/convert-llama-ggmlv3-to-gguf.py --input <path-to-ggml> --output <path-to-gguf>

TODO

• Fix tests
• Move convert script into package to make it easier for people to migrate Add docs link to conversion script in llama.cpp
• Fix detokenization bug (getting extra leading space)

[sndani]

Hello, excited about trying this out with the CodeLlama gguf model.

Followed the MacOS (Sonoma beta) instructions. How do I get the 'llama' shared library?

llama-cpp-python % python3 -m llama_cpp.server --model $MODEL --n_gpu_layers 1
Traceback (most recent call last):
....
File "... /llama-cpp-python/llama_cpp/llama_cpp.py", line 80, in
_lib = _load_shared_library(_lib_base_name)
File " ... /llama-cpp-python/llama_cpp/llama_cpp.py", line 71, in _load_shared_library
raise FileNotFoundError(
FileNotFoundError: Shared library with base name 'llama' not found

Thanks!

[abetlen]

@sndani try reinstalling with the --verbose flag, it's likely a build error and will be reported by cmake. If the issue isn't resolvable please open an issue and I'll take a look. Cheers

[sndani]

@abetlen thanks for the great work and thanks for responding.
Yes, did run
% CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 pip install -U llama-cpp-python --no-cache-dir --verbose

Turns out, cmake isn't getting to target libllama.so under vendor/llama.cpp (but 'make clean' try deleting it). This is a dev fix but will open an issue (or the next person who encounters this can) if this isn't only my environment for some reason.

% make clean
% cd vendor/llama.cpp
% LLAMA_METAL=on make libllama.so
% cd ../..
% pip install 'llama-cpp-python[server]'
% export LLAMA_CPP_LIB=./vendor/llama.cpp/libllama.so
% python3 -m llama_cpp.server --model $MODEL --n_gpu_layers 1

[reddiamond1234]

My model is now a lot slower... is there any solution to fix this?