Compiling SuiteSparse on Windows with CUDA enabled

[cmendestecgraf]

I have recently started integrating SuiteSparse to our in-house geomechanical simulator. Our main compilation environment is on Windows, with Visual Studio 2022. Without CUDA support, compilation was surprisingly easy and everything just worked by providing the desired flags for CMake. I had some difficulties though when I enabled CUDA compilation.

To compile version 7.8.3 on Windows WITH CUDA support, it was necessary to add the following lines to the shared and static libs sections on CHOLMOD's CMakeLists.txt file:

set_target_properties ( CHOLMOD PROPERTIES CUDA_RESOLVE_DEVICE_SYMBOLS ON)
set_target_properties ( CHOLMOD_static PROPERTIES CUDA_RESOLVE_DEVICE_SYMBOLS ON)

Without this, the PerformDeviceLink attribute in the generated vcproj file is set to false, instead of true (found on the GUI at the CHOLMOD_static project properties page, at the "CUDA Linker" section).

This prevents the creation of the needed cholmod_static.device-link.obj file, leading to the following linker error when compiling CHOLMOD's "dl" and "sl" demos (the ones that can use CUDA):

error LNK2019: unresolved external symbol __cudaRegisterLinkedBinary

Turning on the CUDA_RESOLVE_DEVICE_SYMBOLS solves the problem. Notice that this flag is already set to 'on' at the SPQR\GPUQREngine CMakeLists.txt file. The fact that SPQR uses CHOLMOD doesn't seem to be a problem. We where able to build and run SPQR GPU demos.

We have NOT tested if this addition causes any trouble in Linux.

As per the following link, it appears that this is a limitation of CMake itself. See https://gitlab.kitware.com/cmake/cmake/-/issues/17520 for a discussion on the subject.

[mmuetzel]

Thank you for your detailed report and the pointers to past discussions revolving around this topic.

If I understand correctly, the issue arises because CUDA isn't used as the linker for the CHOLMOD libraries.
I've opened #931 with a change that might address that issue.

Could you please check if that fixes the issue for you?

[mmuetzel]

Just out of interest: Which C and CUDA compilers (and versions) are you using?
I wasn't able to reproduce the issue in CI runs...

According to the CI logs, the CI runners are using a pre-installed MS Visual Studio 2022 Enterprise (MSVC 19.43.34808.0), and they install CUDA 12.5.0 (NVIDIA 12.5.40 with host compiler MSVC 19.43.34808.0).

[cmendestecgraf]

Well, I have just tested replacing the CHOLMOD's CMakeLists.txt file with the one provided in #931, but unfortunately it didn't solved the problem. Got the same linker error (LNK2019) as before. I checked, but with the new LINKER_LANGUAGE CUDA option, the cholmod_static.device-link.obj was not generated.

My environment is:
cmake version 3.31.3
-- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.26100.
-- The C compiler identification is MSVC 19.40.33812.0
-- The CXX compiler identification is MSVC 19.40.33812.0
-- The CUDA compiler identification is NVIDIA 12.8.93 with host compiler MSVC 19.40.33812.0

So you have a slightly newer Visual studio version, but an older CUDA toolkit.

Attached are the CMakeLists.txt file that works for me (only tested on Windows) and the log files from running cmake with both my CMakeLists file and the new one. Notice that on the new one, there is no "CudaLink" call to generate the cholmod_static.device-link.obj file. Both of them have the "CudaLink" call to generate gpuqrengine_static.device-link.obj from SPQR.

My command line for running cmake was:
cmake -G"Visual Studio 17 2022" -DCMAKE_INSTALL_PREFIX=.. -DSUITESPARSE_USE_CUDA=ON -DSUITESPARSE_USE_PYTHON=OFF -DSUITESPARSE_USE_FORTRAN=OFF -DSUITESPARSE_C_TO_FORTRAN:STRING="(name,NAME) name" -DBUILD_SHARED_LIBS=OFF -DSUITESPARSE_ENABLE_PROJECTS="suitesparse_config;amd;camd;ccolamd;colamd;cholmod;umfpack;paru;spqr" -DBLA_VENDOR=OpenBLAS -DBLA_STATIC=ON -DBLAS_INCLUDE_DIRS="../../../../../../../ext/blas_lapack/openBlas/include" -DBLAS_LIBRARIES="../../../../../../../ext/blas_lapack/openBlas/libopenblas" -DLAPACK_INCLUDE_DIRS="../../../../../../../ext/blas_lapack/openBlas/include" -DLAPACK_LIBRARIES="../../../../../../../ext/blas_lapack/openBlas/libopenblas" ..

And the build logs where generated by that call to cmake, followed by

msbuild ALL_BUILD.vcxproj /t:Build /p:Configuration=Release /m
msbuild ALL_BUILD.vcxproj /t:Build /p:Configuration=Debug /m
msbuild INSTALL.vcxproj /t:Build /p:Configuration=Release /m

SuiteSparseBuildLog_MyCMakeLists.txt
SuiteSparseBuildLog_NewCMakeLists.txt
CMakeLists.txt

Hope that helps!

[mmuetzel]

Thanks for those build logs.

I'm not used to reading the output of MSVC compilers. But if I understand correctly, the CHOLMOD libraries have been built successfully with the changes from #931.
However, it now fails to link some demos that are built as part of the CHOLMOD subproject.

If I'm understanding the suggestions on the CMake issue tracker that you linked correctly, that happens because you are only building static libraries (otherwise the shared library would be linked to these demos). From what I gather, the preferred option would be to set the CUDA compiler as the linker driver for all libraries and executables that are linking to the static CHOLMOD library.
Ideally, CMake would do that automagically. But it doesn't (at least currently).
We can only do that manually for the binaries that are built as part of SuiteSparse. Other projects that are linking to the static CHOLMOD library (with enabled CUDA) would also need to set the linker driver manually.
That isn't a good solution imho.

If I'm reading that thread correctly, setting CUDA_RESOLVE_DEVICE_SYMBOLS for that static library would work around that issue. That would however mean that no cuda calls (whatever that is) would be possible any more between functions from the static CHOLMOD library and other libraries linking to it.

I don't know what a "cuda call" is. I imagine that are calls in code that is running natively on a GPU (from one library to another.)
As far as I can tell, the API of CHOLMOD only consists of functions that can be called by a CPU. So, setting that property is probably not an issue.
Apparently, the programs are working for you if you built the static CHOLMOD library with CUDA_RESOLVE_DEVICE_SYMBOLS set to ON. So, we could probably do that for the static CHOLMOD library, too.

I'll update #931 with some follow-up changes.

[cmendestecgraf]

Sorry Markus for the late reply. Yesterday I was really busy on several meetings.
Your understanding of the problem seems to be the same as mine. Unfortunately, I'm no CUDA or CMake expert, so I don't have authoritative answers here.
My take is that using CUDA_RESOLVE_DEVICE_SYMBOLS on the cholmod library would only be a problem if the library user wanted to explicitly call the CUDA kernels used by cholmod. Since that is clearly not expected, I think we are good doing that.
With that set to ON, I could link not only the demos but also my SuiteSparse plugin for our in-house simulator.

[mmuetzel]

A CI runner that built only static libraries with CUDA succeeded with additional changes in #931:
https://github.com/DrTimothyAldenDavis/SuiteSparse/actions/runs/14066043881/job/39388668673?pr=931

That could still be because it is using a different (older) version of CUDA than you.
Could you please check if the changes are working for you? (But since they are setting CUDA_RESOLVE_DEVICE_SYMBOLS to ON like you suggested, I'm guessing they will.)

The only remaining question might be whether that should only be done for MSVC (and MSVC-compatible?) compilers.
Do you have access to clang-cl or icx? Do they also need that property set for static libraries?
Are there (actual) disadvantages if that property is set on other platforms/compilers?

(And only slightly related: Should we check if these properties are set reasonably for the other CUDA libraries (e.g., in SPQR)?)

[cmendestecgraf]

Just tested the new CMakLists.txt file and the compilation is fine now. Thank you very much for the effort!

At the moment I don't have access to either Clang or the Intel compiler, but we intend to try Clang in the future to do some performance tests. So, I'm sorry to not being able to answer your questions about the other environments.

As for SPQR, it seems to be compiling ok with the current files, but notice that SPQR CMakLists.txt does have the CUDA_RESOLVE_DEVICE_SYMBOLS clause set to ON. I didn't integrated it with our application yet (just Cholmod and Umfpack), but I can confirm that I was able to compile, link and run SPQR's demo programs.