Compile llama.cpp with CUDA 12

[artyomboyko]

I am building llamacpp with CUDA 12 support (RTX5000). How can I add support for RTX4000 and RXT5000 using cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=“86;89”? The range of acceptable values for the DCMAKE_CUDA_ARCHITECTURES parameter is not specified in documentation. Should I specify CUDA 12 as 120 or as 12? For example:

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=”86;89;120”

[0cc4m]

Those are cuda compute capabilities, basically device architectures, not cuda versions. See https://developer.nvidia.com/cuda-gpus

[artyomboyko]

@0cc4m OK. How do I set this parameter correctly so that RTX4000 and RTX5000 can be used?

[CommanderLake]

Dont modify CMAKE_CUDA_ARCHITECTURES the llama.cpp CUDA backend should run on newer GPUs without explicitly setting the architectures.

[artyomboyko]

@CommanderLake I will try, thanks. I'll come back with feedback.