upgrade to use the latest librdkafka

[leiwang008]

Hi shasha,
I know this project is not maintained, but you are the founder of this project and have many knowledge and experiences :-) . I would like to know if you have any idea about the problem I met, I appreciate any help.

I would like to use the "oauthbearer/oidc" feature provided by the librdkafka2.4.0 in our Unreal Project on platform Windows x86_64.
I cloned your project and then I upgraded EasyKafka to use the https://github.com/morganstanley/modern-cpp-kafka/releases/tag/v2024.07.03 and I could get it built successfully.
Then I tried to upgrade the librdkafka to use the version 2.4.0, the following is what I did

1. I got the librdkafka2.4.0 source code from https://github.com/confluentinc/librdkafka/releases/tag/v2.4.0
2. Go to the win32 folder and open the librdkafka.sln with the Visual Studio 2022
3. Then try to build it by "Build->Build Solution", this will download a lot of dependencies
4. There will be some build errors "The build tools for v143 (Platform Toolset = 'v143') cannot be found. To build using the v143 build tools, please install v143 build tools." for some projects (the examples, tests projects etc.), the main c-project librdkafka got built successfully and that is what we need, so I just ignore those build errors.
5. By default, the build will be for "win32", we need "X64 build", go to "Build->Config Manager ...", change the "Active solution platform" to "x64" and change the "Active solution configuration" to "Release"
6. By default, the build will be "Dynamic Library" (.dll), we porbably need a "Static Libary" (everything in one .lib file), right-click the "librdkafak" and select "Properties", and go to "General -> Configuration Type" and change it to "Static library (.lib)".
7. Then right-click the project "librdkafka" and select "ReBuild", and we will get the release build "librdkafka.lib" in folder like "win32\outdir\v143\x64\Release", copy this "librdkafka.lib" to "UE-EasyKafka\Source\ThirdParty\KafkaLib\lib\Win64" and modify "KafkaLib.Build.cs" to use this library instead of the old one.
8. Then we do the same thing for project 'librdkafkacpp' to build and get static library librdkafkacpp.lib, and replace the one in EasyKafka plugin.
9. When I build the SASConnector plugin code, I got the following errors
   Severity Code Description Project File Line Suppression State Details
   Error LNK2001 unresolved external symbol __imp_crc32 LyraStarterGame C:\repository\lyra-demo-kafka\LyraStarterGame\Intermediate\ProjectFiles\librdkafka.lib(rdkafka_admin.obj) 1
   So I checked the "librdkafka" solution, I found that it depends some installed libraries which have been put in C:\sdk\librdkafka-2.4.0\vcpkg_installed\x64-windows\x64-windows\lib (libcurl.lib, libcrypto.lib, libssl.lib, zlib.lib,zstd.lib etc.) and copied them to the "UE-EasyKafka\Source\ThirdParty\KafkaLib\lib\Win64" and modified the KafkaLib.Build.cs to include them
10. After all these modifications, I successfully built the whole project
11. But when I try to launch the project, it says that the module "EasyKafka" cannot be loaded, "Plugin 'EasyKafka' failed to load because module 'EasyKafka' could not be loaded. There may be an operating system error or the module may not be properly set up."
12. I noticed that there are some .dll files (libcrypto-3-x64.dll, libssl-3-x64.dll, zlib1.dll, libcurl.dll, zstd.dll) in the folder C:\repository\github\librdkafka\vcpkg_installed\x64-windows\x64-windows\bin, I copied them to the folder "UE-EasyKafka\Source\ThirdParty\KafkaLib\bin\Win64" and then include them as below

            string binPath = Path.Combine(ModuleDirectory, "bin/Win64");
            RuntimeDependencies.Add(Path.Combine(binPath, "libcrypto-3-x64.dll"));
            RuntimeDependencies.Add(Path.Combine(binPath, "libssl-3-x64.dll"));
            RuntimeDependencies.Add(Path.Combine(binPath, "zlib1.dll"));
            RuntimeDependencies.Add(Path.Combine(binPath, "libcurl.dll"));
            RuntimeDependencies.Add(Path.Combine(binPath, "zstd.dll"));

but this no luck, cannot load the EasyKafka plugin.

[leiwang008]

Hi,
I just got it done. I built the librdkafka into a static library and also got the dependencies as static library. Then everything works amazingly. By the way, I also upgrade to use the latest modern-cpp-kafka for this EasyKafka Plugin. If you like I can contribute my code to the project :-).

[sha3sha3]

Hey @leiwang008
Three rules of thumb :B

1. use static libraries with UE, it will save you some tears.
2. If you want to support linux, make sure to use libc++ and not libstdc++ (librdkafka is c library as far as i can remember so doesn't matter).
3. Always compile against ur UE thirdparty libraries (Like OpenSSL, ZLib) - If you are using dynamic linked library its a must.

For Windows VCPKG is really handy
For Linux you can use the cross compile toolchain from EPIC (clang), or even compiling on native Linux again using CLANG (Remember Libc++ :) ).

[leiwang008]

Hi @sha3sha3
Thank you very much for the detail explaination 👍
We just need to get it run on the Windows for now, no need for Linux.

I did run into the conflict problem for the (OpenSSL, ZLib), I fixed it by commenting out the UE library and use the static libraries (built from librdkafka) instead, do you think this is a problem.

        string LibPath = Path.Combine(ModuleDirectory, "lib/Win64");
        PublicAdditionalLibraries.Add(Path.Combine(LibPath, "lz4.lib"));
        PublicAdditionalLibraries.Add(Path.Combine(LibPath, "rdkafka.lib"));
        PublicAdditionalLibraries.Add(Path.Combine(LibPath, "rdkafka++.lib"));

        PublicAdditionalLibraries.Add(Path.Combine(LibPath, "libcrypto.lib"));
        PublicAdditionalLibraries.Add(Path.Combine(LibPath, "libcurl.lib"));
        PublicAdditionalLibraries.Add(Path.Combine(LibPath, "libssl.lib"));
        PublicAdditionalLibraries.Add(Path.Combine(LibPath, "zlib.lib"));
        PublicAdditionalLibraries.Add(Path.Combine(LibPath, "zstd.lib"));

        // comment out the following, there are conflicts between libcrypto.lib/libssl.lib(Version: 3.0.8)
        // and UE5'S OpenSSL(C:\Program Files\Epic Games\UE_5.3\Engine\Binaries\ThirdParty\OpenSSL)
       /* PublicDependencyModuleNames.AddRange(
        new string[]
        {
            "OpenSSL",
            "zlib"
        }
        );*/

[sha3sha3]

Hey @leiwang008

The problem is that you didn't compile Librdkafka with UE dependencies.

Your solution isn't practical.
The moment you use any module dependent on OpenSSL you will face the same issue.

Unreal engine uses OpenSSL 1.1x, instead you compiled the Library with OpenSSL 3.x.

[leiwang008]

Hi @sha3sha3
Thanks for the valuable information, I used to be a Java developer and I am new to c/c++ and Unreal Engine. How can I build the librdkafka with UE dependencies? I am using Visual Studio 2022 to build the librdkafka. There is too few information related to UE on the google. Are UE dependencies in the folder C:\Program Files\Epic Games\UE_5.3\Engine\Binaries? I found there is a folder C:\Program Files\Epic Games\UE_5.3\Engine\Binaries\ThirdParty\OpenSSL\Win64\VS2015 holding 2 .dll files libeay32.dll and ssleay32.dll? Are they the UE dependencies that I should use?
When I built the librdkafka, it seems that the visual studio will download automatically the dependencies into the folder
"... librdkafka\vcpkg_installed\x64-windows-static\x64-windows-static\lib", if I build the librdkafka against the UE denpendencies, which dependencies it will use? the UE denpendencies or the dependencies in "... librdkafka\vcpkg_installed\x64-windows-static\x64-windows-static\lib"?

[leiwang008]

Ok, I found the UE libraries
C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\OpenSSL\1.1.1t
C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\libcurl\8.4.0
C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\zlib\1.2.13

When I built librdkafka, it will downloaded automatically the libraries of following version
openssl: 3.0.8
libcurl: 7.86.0-DEV
zlib: 1.2.13

The libraries versions are not consistent except the "zlib".

I tried to build the librdkafka with EU provided libraries (OpenSSL\1.1.1t, ibcurl\8.4.0, zlib\1.2.13), I used the visual studio and I did this by add the "include" folders to "Include Directories" in the VC++ Directories section
C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\zlib\1.2.13\include
C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\OpenSSL\1.1.1t\include\Win64\VS2015
C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\libcurl\8.4.0\include

I also add the library folders to "Library Directories" in the VC++ Directories section
C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\libcurl\8.4.0\lib\Win64\Release
C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\OpenSSL\1.1.1t\lib\Win64\VS2015\Release
C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\zlib\1.2.13\lib\Win64\Release

I could build it successfully and I got the librdkafka.lib and librdkafkacpp.lib, I copied them to the "UE-EasyKafka\Source\ThirdParty\KafkaLib\lib\Win64" and modified the KafkaLib.Build.cs to include them and use the UE libraries as below

 string LibPath = Path.Combine(ModuleDirectory, "lib/Win64");
 PublicAdditionalLibraries.Add(Path.Combine(LibPath, "lz4.lib"));
 PublicAdditionalLibraries.Add(Path.Combine(LibPath, "librdkafka.lib"));
 PublicAdditionalLibraries.Add(Path.Combine(LibPath, "librdkafkacpp.lib"));
 PublicAdditionalLibraries.Add(Path.Combine(LibPath, "zstd.lib"));

 PublicDependencyModuleNames.AddRange(
 new string[]
 {
     "OpenSSL",
     "zlib",
     "libcurl"
 }
 );

But it gave me a lot of errors
Severity Code Description Project File Line Suppression State Details
Error LNK2019 unresolved external symbol __imp_curl_easy_cleanup referenced in function rd_http_get connector C:\repository\github\connector\Intermediate\ProjectFiles\librdkafka.lib(rdhttp.obj) 1
Error LNK2019 unresolved external symbol __imp_curl_easy_cleanup referenced in function rd_http_get connector C:\repository\github\connector\Intermediate\ProjectFiles\librdkafka.lib(rdhttp.obj) 1
Error LNK2019 unresolved external symbol __imp_curl_easy_cleanup referenced in function rd_http_get connector C:\repository\github\connector\Intermediate\ProjectFiles\librdkafka.lib(rdhttp.obj) 1
Error LNK2019 unresolved external symbol __imp_curl_easy_cleanup referenced in function rd_http_get connector C:\repository\github\connector\Intermediate\ProjectFiles\librdkafka.lib(rdhttp.obj) 1

I noticed that the library size are different between the UE's and automatically downloaded version, for example libcurl.lib, UE's version is 2180KB, but automatically downloaded version is much bigger, it is 12531KB

Do you have any idea about these errors, any suggestions?