Commit `7c898d5` breaks generation on GPU

[iamlemec]

From commit 7c898d onwards, the output of any type of generation/completion on the GPU is just "#" repeated forever. For instance, using the example from README.md

from llama_cpp import Llama
llm = Llama(model_path='models/llama2-7b.q4_0.gguf', n_gpu_layers=100)
for s in llm('Building a website can be done in 10 simple steps:\nStep 1:', stream=True):
    print(s)

The output is the following repeated:

{'id': 'cmpl-14ed3b80-49af-453d-99a4-c7925f5680f7', 'object': 'text_completion', 'created': 1705351368, 'model': 'models/llama2-7b.q4_0.gguf', 'choices': [{'text': '#', 'index': 0, 'logprobs': None, 'finish_reason': None}]}

Generation works fine on the CPU and for previous commits. Doesn't seem to be related to quantization or model type. Interestingly, generation also works using pure llama.cpp through the main interface for both CPU and GPU. I tested this out for the current master and the commits around the above change (notably 76484fb and 1d11838). I also managed to get it working in llama-cpp-python using the low level API, just using simple batching and llama_decode.

Environment info:

GPU: RTX A6000
OS: Linux 6.6.0-0.rc5
CUDA SDK: 12.2
CUDA Drivers: 535.113.01

Thanks!

[abetlen]

@iamlemec will investigate

[markyfsun]

Same here.

llama_cpp_python: 0.2.29
GPU: NVIDIA 3090
OS: Ubuntu 22.04
CUDA SDK: 12.2
CUDA Drivers: 535.146.02

I am using the fastapi server. I observed that the server could generate meaningful response at first a few short inputs. When I asked it to response to a long input, it repeated # forever. Then I retried with previous short inputs, I got only #.

Downgrade llama_cpp_python to 0.2.28 solves the issue.

[abetlen]

Hmmm, have not been able to reproduce this on my Nvidia 2060 card with q4_k_m quants.

[xmaayy]

Hmmm, have not been able to reproduce this on my Nvidia 2060 card with q4_k_m quants.

I have a reproduction example on the other issue #1091 (comment)

[iamlemec]

I think I found the answer! You need to set offload_kqv=True for things to work. The default in the Llama class is False, but the underlying default from llama_context_default_params is True, which explains why it was working with the low level API.

[abetlen]

@iamlemec yup that broke it for me too, great catch. Can you try it out with llama.cpp with --no_kv_offload, if it's present there then we can open an upstream issue and get it fixed!

[iamlemec]

Yup, getting all #s from the same prompt as above using --no_kv_offload! Do you want to take the lead on the upstream bug or shall I?

[slaren]

This in indeed a bug in llama.cpp, but I would strongly recommend enabling offload_kqv by default, as it is in llama.cpp. Even in cases with low VRAM, it is usually better to offload less layers and keep offload_kqv enabled.

[m-from-space]

I can confirm that version 0.2.29 breaks generation completely for different models.

[iactix]

This in indeed a bug in llama.cpp, but I would strongly recommend enabling offload_kqv by default, as it is in llama.cpp. Even in cases with low VRAM, it is usually better to offload less layers and keep offload_kqv enabled.

I seem to end up with 1.1 tokens per second instead of 1.5 after enabling the flag and adjusting the layers to refit everything into VRAM. It is a really bad setup (2600X & RTX1080) running nous-hermes-2-yi-34b.Q4_K_M.gguf with now 14 instead of 18 layers offloaded to GPU. Just providing this as a data point.