Issues with running Llama.cpp on Raspberry Pi 5 with Vulkan.

[Raewari]

Hey I'm really excited about this new Vulkan support. I'm having issues getting it work work on my Raspberry Pi 5 running RPI OS.
I've installed all the required mesa-vulkan drivers & dev tools beforehand.

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make LLAMA_VULKAN=1 -j4
wget https://huggingface.co/jartine/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/TinyLlama-1.1B-Chat-v1.0.Q4_0.gguf
./main -e -m ~/TinyLlama-1.1B-Chat-v1.0.Q4_0.gguf -p '# Famous Speech\nFour score and' -n 50

#2059
This results in a deadlock/Model refusing to run, any help would be appreciated.

[gurgalof]

It hangs at this point for me. (VK_DEBUG=1)

ggml_vk_init()
ggml_vk_find_queue_family_index()
ggml_vk_find_queue_family_index()
ggml_vulkan: Using V3D 7.1.7 | uma: 1 | fp16: 0 | warp size: 16
ggml_vk_load_shaders()
ggml_vk_create_pipeline(matmul_f32_l, main, 3, 56, (128,128,1), specialization_constants, 1)
ggml_vk_create_pipeline(matmul_f32_m, main, 3, 56, (64,64,1), specialization_constants, 1)
ggml_vk_create_pipeline(matmul_f32_s, main, 3, 56, (32,32,1), specialization_constants, 1)
ggml_vk_create_pipeline(matmul_f32_aligned_l, main, 3, 56, (128,128,1), specialization_constants, 128)
ggml_vk_create_pipeline(matmul_f32_aligned_m, main, 3, 56, (64,64,1), specialization_constants, 64)
ggml_vk_create_pipeline(matmul_f32_aligned_s, main, 3, 56, (32,32,1), specialization_constants, 32)
ggml_vk_create_pipeline(matmul_f16_l, main, 3, 56, (128,128,1), specialization_constants, 1)
ggml_vk_create_pipeline(matmul_f16_m, main, 3, 56, (64,64,1), specialization_constants, 1)
ggml_vk_create_pipeline(matmul_f16_s, main, 3, 56, (32,32,1), specialization_constants, 1)
ggml_vk_create_pipeline(matmul_f16_aligned_l, main, 3, 56, (128,128,1), specialization_constants, 128)
ggml_vk_create_pipeline(matmul_f16_aligned_m, main, 3, 56, (64,64,1), specialization_constants, 64)
ggml_vk_create_pipeline(matmul_f16_aligned_s, main, 3, 56, (32,32,1), specialization_constants, 32)
ggml_vk_create_pipeline(matmul_f16_f32_l, main, 3, 56, (128,128,1), specialization_constants, 1)
ggml_vk_create_pipeline(matmul_f16_f32_m, main, 3, 56, (64,64,1), specialization_constants, 1)
ggml_vk_create_pipeline(matmul_f16_f32_s, main, 3, 56, (32,32,1), specialization_constants, 1)
ggml_vk_create_pipeline(matmul_f16_f32_aligned_l, main, 3, 56, (128,128,1), specialization_constants, 128)
ggml_vk_create_pipeline(matmul_f16_f32_aligned_m, main, 3, 56, (64,64,1), specialization_constants, 64)
ggml_vk_create_pipeline(matmul_f16_f32_aligned_s, main, 3, 56, (32,32,1), specialization_constants, 32)
ggml_vk_create_pipeline(f32_to_f16, main, 2, 16, (64,1,1), specialization_constants, 1)
ggml_vk_create_pipeline(dequant_f16, main, 2, 16, (8192,1,1), specialization_constants, 1)
ggml_vk_create_pipeline(dequant_q4_0, main, 2, 16, (8192,1,1), specialization_constants, 1)
ggml_vk_create_pipeline(dequant_q4_1, main, 2, 16, (8192,1,1), specialization_constants, 1)
ggml_vk_create_pipeline(dequant_q5_0, main, 2, 16, (8192,1,1), specialization_constants, 1)
ggml_vk_create_pipeline(dequant_q5_1, main, 2, 16, (8192,1,1), specialization_constants, 1)
ggml_vk_create_pipeline(dequant_q8_0, main, 2, 16, (8192,1,1), specialization_constants, 1)
ggml_vk_create_pipeline(dequant_q2_K, main, 2, 16, (8192,1,1), specialization_constants, 1)
ggml_vk_create_pipeline(dequant_q3_K, main, 2, 16, (8192,1,1), specialization_constants, 1)
ggml_vk_create_pipeline(dequant_q4_K, main, 2, 16, (8192,1,1), specialization_constants, 1)
ggml_vk_create_pipeline(dequant_q5_K, main, 2, 16, (8192,1,1), specialization_constants, 1)

Raspberry Pi 5 running Arch Linux ARM

[0cc4m]

Thanks for the detailed report. It's not a good sign that it doesn't even get past loading the shaders. Probably rather difficult to figure out the problem here. Any driver messages in the system log?

[gurgalof]

Nothing in the system log.
Is there anything else I can do to help diagnose this issue?

[0cc4m]

Not without delving into C++ and Vulkan code, I think. I have a Pi 4 somewhere, maybe I'll try it sometime. But it's not high priority for now. Would be really cool if it worked though.

[pure-water]

Are you sure Raspberry Pi is Vulkan Compatible at all？

[gurgalof]

Yes, it is Vulkan 1.2 Conformant.

[gurgalof]

I got it to get past loading the shaders by running with V3D_DEBUG=noloopunroll ./main -m model... etc.
It instead fails on:

ggml_vk_create_buffer(1, { HostVisible | HostCoherent | HostCached })
ggml_vulkan: Failed to allocate pinned memory.
ggml_vulkan: No suitable memory type found: ErrorOutOfDeviceMemory

vulkaninfo output for memory:

VkPhysicalDeviceMemoryProperties:
=================================
memoryHeaps: count = 1
	memoryHeaps[0]:
		size   = 4294967296 (0x100000000) (4.00 GiB)
		budget = 4294967296 (0x100000000) (4.00 GiB)
		usage  = 0 (0x00000000) (0.00 B)
		flags: count = 1
			MEMORY_HEAP_DEVICE_LOCAL_BIT
memoryTypes: count = 1
	memoryTypes[0]:
		heapIndex     = 0
		propertyFlags = 0x0007: count = 3
			MEMORY_PROPERTY_DEVICE_LOCAL_BIT
			MEMORY_PROPERTY_HOST_VISIBLE_BIT
			MEMORY_PROPERTY_HOST_COHERENT_BIT
		usable for:
			IMAGE_TILING_OPTIMAL:
				color images
				FORMAT_D16_UNORM
				FORMAT_X8_D24_UNORM_PACK32
				FORMAT_D32_SFLOAT
				FORMAT_D24_UNORM_S8_UINT
				(non-sparse)
			IMAGE_TILING_LINEAR:
				color images
				(non-sparse)

It has no HostCached flag so I did the same as #5319
This got it working (I think) but its orders of magnitude slower than running on CPU.
Still would need to run tests etc, but this being so slow I feel it's not worth it.

It getting past the shader loading with V3D_DEBUG=noloopunroll got me thinking that it wasn't a spinlock before but maybe extremly slow loop unrolling when loading shaders.
And yep, without the environment variable it gets past loading the shaders after a very long time.

[0cc4m]

Shaders can take a little bit to load, but they should get cached and load basically instantly the second time. Is it coherent? I don't think my matmul shader is correct for warp size 16 yet.

[gurgalof]

Yes, the second time it loads instantly. First time though maybe up to 30min (I left for a bit).
I didn't check the output if it's coherent or not but I can try again a bit later.

[gurgalof]

./main -m tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf -ngl 100 -p '# Famous Speech\nFour score and' -n 50
15min+ later and no it is not coherent (it's pretty much garbled).

[0cc4m]

Then it probably needs at the very least a fixup for those shaders that assume a warp size of 32/64. But also some optimization or it's not worth using over CPU. I'm not sure if that's possible.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.