Freeze after offloading layers to GPU

[CRD716]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

llama.cpp does not freeze and continues to run normally, not interfering with basic windows operations.

Current Behavior

llm_load_print_meta: arch           = llama
llm_load_print_meta: vocab type     = SPM
llm_load_print_meta: n_vocab        = 32000
llm_load_print_meta: n_merges       = 0
llm_load_print_meta: n_ctx_train    = 4096
llm_load_print_meta: n_ctx          = 4096
llm_load_print_meta: n_embd         = 8192
llm_load_print_meta: n_head         = 64
llm_load_print_meta: n_head_kv      = 8
llm_load_print_meta: n_layer        = 80
llm_load_print_meta: n_rot          = 128
llm_load_print_meta: n_gqa          = 8
llm_load_print_meta: f_norm_eps     = 1.0e-05
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: n_ff           = 28672
llm_load_print_meta: freq_base      = 10000.0
llm_load_print_meta: freq_scale     = 1
llm_load_print_meta: model type     = 70B
llm_load_print_meta: model ftype    = mostly Q5_K - Medium
llm_load_print_meta: model size     = 68.98 B
llm_load_print_meta: general.name   = LLaMA v2
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: LF token  = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.23 MB
llm_load_tensors: using CUDA for GPU acceleration
llm_load_tensors: mem required  = 35995.03 MB (+ 1280.00 MB per state)
llm_load_tensors: offloading 18 repeating layers to GPU
llm_load_tensors: offloaded 18/83 layers to GPU
llm_load_tensors: VRAM used: 10500 MB

llama.cpp then freezes and will not respond. Task Manager shows 0% CPU or GPU load. It is also somehow unable to be stopped via task manager, requiring me to hard reset my computer to end the program. It also causes general system instability, as I am writing this with my desktop blacked out and file explorer frozen.

Environment and Context

Windows 10
128 GB RAM
Threadripper 3970X
RTX 2080TI
CMake 3.27.4
CUDA 12.2

Failure Information (for bugs)

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

Run a model with CUBlas.
(My exact command: main -ngl 18 -m E:\largefiles\LLAMA-2\70B\uni-tianyan-70b.Q5_K_M.gguf --color -c 4096 --temp 0.6 --repeat_penalty 1.1 -n -1 --interactive-first)

Failure Logs

I'd love to attach them, but file manager stopped working. I'll try and run it again tomorrow and upload the log before everything freezes.

[KerfuffleV2]

How much RAM do you have? Running a Q5_K_M 70B model is going to require around 64GB RAM and pretty much nothing else running on your system. Also, at startup it's going to need to read about 50GB from storage. If you're using a hard drive, this can easily take several minutes.

It also causes general system instability

If a user application can do this, that's basically a problem with the OS.

[CRD716]

How much RAM do you have? Running a Q5_K_M 70B model is going to require around 64GB RAM and pretty much nothing else running on your system. Also, at startup it's going to need to read about 50GB from storage. If you're using a hard drive, this can easily take several minutes.

It also causes general system instability

If a user application can do this, that's basically a problem with the OS.

I have 128 GB of ram, and roughly 2-3 commits ago it loaded within a decent amount of time. Also I don't disagree with the OS problem comment but I have a tendency to mess up linux installs.

[CRD716]

80% sure #3110 is the commit that borked it. https://github.com/ggml-org/ci/tree/results/llama.cpp/d5/4a4027a6ebda98ab0fef7fa0c2247d0bef132a/ggml-4-x86-cuda-v100

[JohannesGaessler]

80% sure #3110 is the commit that borked it.

Did you actually track down the exact commit that caused the issue? Did you test whether or not you still get the issue with -ngl 0 or when compiling without cuBLAS entirely?

[CRD716]

80% sure #3110 is the commit that borked it.

Did you actually track down the exact commit that caused the issue? Did you test whether or not you still get the issue with -ngl 0 or when compiling without cuBLAS entirely?

Sorry, just recently had time to test. I've updated to the latest version and it gets further in the process before just crashing. It does this both with and without -ngl. The log cuts off after "warming up the model with an empty run"

[1694833318] Log start
[1694833318] Cmd: main -m E:\largefiles\LLAMA-2\70B\marcoroni-70b.Q5_K_S.gguf --mirostat 2 --color -c 4096 --temp 0.6 --repeat_penalty 1.1 -n -1 --interactive-first -ins
[1694833318] main: build = 1247 (e6616cf)
[1694833318] main: built with  for unknown
[1694833318] main: seed  = 1694833318
[1694833318] main: llama backend init
[1694833319] main: load the model and apply lora adapter, if any
[1694833323] warming up the model with an empty run

[CRD716]

llm_load_print_meta: format         = GGUF V2 (latest)
llm_load_print_meta: arch           = llama
llm_load_print_meta: vocab type     = SPM
llm_load_print_meta: n_vocab        = 32000
llm_load_print_meta: n_merges       = 0
llm_load_print_meta: n_ctx_train    = 4096
llm_load_print_meta: n_ctx          = 4096
llm_load_print_meta: n_embd         = 8192
llm_load_print_meta: n_head         = 64
llm_load_print_meta: n_head_kv      = 8
llm_load_print_meta: n_layer        = 80
llm_load_print_meta: n_rot          = 128
llm_load_print_meta: n_gqa          = 8
llm_load_print_meta: f_norm_eps     = 1.0e-05
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: n_ff           = 28672
llm_load_print_meta: freq_base      = 10000.0
llm_load_print_meta: freq_scale     = 1
llm_load_print_meta: model type     = 70B
llm_load_print_meta: model ftype    = mostly Q5_K - Small
llm_load_print_meta: model size     = 68.98 B
llm_load_print_meta: general.name   = LLaMA v2
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: LF token  = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.23 MB
llm_load_tensors: using CUDA for GPU acceleration
llm_load_tensors: mem required  = 38529.47 MB (+ 1280.00 MB per state)
llm_load_tensors: offloading 12 repeating layers to GPU
llm_load_tensors: offloaded 12/83 layers to GPU
llm_load_tensors: VRAM used: 6733 MB
....................................................................................
E:\Installs\llama.cpp\build\bin\Release>

Here's the command prompt output

[Green-Sky]

did you try supplying a prompt, either with -p or -f ?

[CRD716]

did you try supplying a prompt, either with -p or -f ?

Yes, same issue.

[Void-025]

I'm having the same issue on EndeavourOS with an RX580 and hipBLAS, it just freezes after the section where it prints dots. Using it with no layers on the GPU works fine for me, though.

[Void-025]

Just tried it with CLBlast and that works properly, though very slowly, has anyone figured out what causes cuBLAS and hipBLAS to freeze?

[8XXD8]

With hipBlas I can only load large models with --no-mmap, otherwise it just loads forever.

[Void-025]

Just tried that, seems to still get stuck after printing a bunch of dots.

[8XXD8]

You need a ton of ram, or swap for it to work. I could not load a 70b q3_s model with -no-mmap on a 32gb vram 32gb ram machine, but with 40gb swap it works with 32gb vram and 16b ram.

[Void-025]

It's a 7b model, which I've been able to load with CLBlast, and I'm only offloading 1 layer anyway (for now, to test if it works).