Segmentation fault in libaws-cpp-sdk-core.so on creation of Aws::Client::ClientConfiguration

[gi12345]

Describe the bug

Segmentation fault in libaws-cpp-sdk-core.so on creation of Aws::Client::ClientConfiguration.

Stack trace and reproduction steps see below.

Expected Behavior

No segmentation fault

Current Behavior

Crash Trace:

03/05/2022 15:57:58.394 [WARNING]Catch dump signal [11]Segmentation fault
Obtained 10 stack frames:
[0]./VIGILCameraServer(_Z4dumpi+0x28) [0x4e4cb8]
[1]linux-vdso.so.1(__kernel_rt_sigreturn+0) [0x7f89faf5e4]
[2]/mnt/plugin/vigilcamera/lib/libaws-cpp-sdk-core.so(_ZN3Aws5Utils9Threading16ReaderWriterLock10LockReaderEv+0) [0x7f897f14a8]
[3]/mnt/plugin/vigilcamera/lib/libaws-cpp-sdk-core.so(_ZNK3Aws6Config32ConfigAndCredentialsCacheManager9GetConfigERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_+0x34) [0x7f898017d4]
[4]/mnt/plugin/vigilcamera/lib/libaws-cpp-sdk-core.so(_ZN3Aws6Config20GetCachedConfigValueERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x58) [0x7f89801958]
[5]/mnt/plugin/vigilcamera/lib/libaws-cpp-sdk-core.so(_ZN3Aws6Client38setLegacyClientConfigurationParametersERNS0_19ClientConfigurationE+0x2a0) [0x7f89801c40]
[6]/mnt/plugin/vigilcamera/lib/libaws-cpp-sdk-core.so(_ZN3Aws6Client19ClientConfigurationC1Ev+0x114) [0x7f89802204]
[7]./VIGILCameraServer(_ZN15CMotionUploaderC1Ev+0x44) [0x59713c]
[8]./VIGILCameraServer(_ZN18CVIGILCameraServer12CreateServerEv+0xb7c) [0x4e3364]
[9]./VIGILCameraServer(_ZN18CVIGILCameraServer10InitializeEv+0x64) [0x4e2134]

Reproduction Steps

---- MotionUploader.h file ---------------------

using namespace std;
class CMotionUploader : public Runable
{
public:
	CMotionUploader();
	~CMotionUploader();

public:
	virtual bool run();

private:
    Aws::SDKOptions m_awsSDKoptions;
    Aws::Client::ClientConfiguration m_cfg;
}

---- MotionUploader.cpp file --------------------

CMotionUploader::CMotionUploader()
{
    Aws::InitAPI(m_awsSDKoptions);
}

main()
{
CMotionUploader*    m_pMotionUploader;
m_pMotionUploader = new CMotionUploader();
}

Possible Solution

No response

Additional Information/Context

This code has been working before running daily for a couple of years. The AWS SDK used was about 2018.
The bug is most likely triggered by a design change inside AWS-SDK.
See also likely similar issue, but not for C++: Segmentation fault at Aws::Utils::Threading::ReaderWriterLock::LockReader () #1716

AWS CPP SDK version used

AWS-SDK-C++ version: new checkout and default build "S3-only" on April 18, 2022 OS: Built on Ubuntu-20 AARM64, run on AARM64-device

Compiler and Version used

Ubuntu20-gcc

Operating System and version

Ubuntu-20 AARM64

[vudh1]

Hi @gi12345 can you send the cmake command line that you used to build?

[gi12345]

This is part of a very large system. We do not use cmake, all is built through make.

[gi12345]

Also, look at this one:
Segmentation fault at Aws::Utils::Threading::ReaderWriterLock::LockReader () #1716
#1716

Very similar to this case I think.
But in our case I cannot change the change the initialization order. On creation of class CMotionUploader the c++ compiler will first create the class variable Aws::Client::ClientConfiguration m_cfg before calling the class constructor with Aws::InitAPI(m_awsSDKoptions).

[vudh1]

You are setting the ClientConfiguration outside InitAPI(). I change the code and it works without showing the Segmentation problem. Please follow this code snippet or reopen if this is not helpful:

CMotionUploader() {
        InitAPI(options);
        {
            Aws::Client::ClientConfiguration clientConfig;
        }
    }

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[gi12345]

Hi Daniel,
The solution you suggest is not addressing our problem:

1. We are working in C++ not C; in this specific case C is trivial, C++ is not (see solution to Segmentation fault at Aws::Utils::Threading::ReaderWriterLock::LockReader () #1716 above). Also take into account that the entire scenario happens during class instance creation mainly performed by the C++ compiler. And as in Segmentation fault at Aws::Utils::Threading::ReaderWriterLock::LockReader () #1716 above our code was working before with an older SDK version.
2. The definition Aws::Client::ClientConfiguration m_cfg is a class member definition. It must be a class member definition so class methods can access this data structure! In your suggestion above client_config is a local method variable (scope); we must have a class variable so access from other class methods is possible.
3. The method Aws::InitAPI(m_awsSDKoptions) is run inside the class constructor. This is the earliest time during C++ class creation when a user defined method can be called.

---

In my opinion the problem is caused by Aws::Client::ClientConfiguration m_cfg. I guess it is bound to an AWS internal constructor which runs prior to Aws::InitAPI(m_awsSDKoptions). C++ compiler performs class member variable handling prior to invoking class constructor method.

If the class member variable definition Aws::Client::ClientConfiguration m_cfg would automatically call Aws::InitAPI() the problem would be solved.
An other solution might be the definition of some "dummy" class variable which has a constructor bound and invokes Aws::InitAPI(); such "dummy" variable would be added to the class variable section prior to Aws::Client::ClientConfiguration m_cfg.

[gi12345]

Hi Daniel,
please re-open the issue as I do not consider it solved.

[gi12345]

As an experiment I have modified my code as follows:
...within C++ class constructor...

    printf("CSMU CMotionUploader::GetSecurityKeys: Point AAAAA1\n");
    Aws::SDKOptions k_awsSDKoptions;
    Aws::InitAPI(k_awsSDKoptions);
    printf("CSMU CMotionUploader::GetSecurityKeys: Point AAAAA2\n");
    {
    Aws::Client::ClientConfiguration m_cfg;  //xxx-gi
    printf("CSMU CMotionUploader::GetSevcurityKeys: Point AAAAA3\n");

Ouput:

CSMU CMotionUploader::GetSecurityKeys: Point AAAAA1
CSMU CMotionUploader::GetSecurityKeys: Point AAAAA2

Crash stack trace:

24/05/2022 14:25:56.101 [WARNING]Catch dump signal [11]Segmentation fault

Obtained 10 stack frames:

[0]./VIGILCameraServer(_Z4dumpi+0x28) [0x4e4dc4]
[1]linux-vdso.so.1(__kernel_rt_sigreturn+0) [0x7fafb455e4]
[2]/mnt/plugin/vigilcamera/lib/libaws-cpp-sdk-core.so(_ZNK3Aws8Internal21AWSHttpResourceClient34GetResourceWithAWSWebServiceResultB5cxx11ERKSt10shared_ptrINS_4Http11HttpRequestEE+0xa4) [0x7faf349d7c]
[3]/mnt/plugin/vigilcamera/lib/libaws-cpp-sdk-core.so(_ZNK3Aws8Internal17EC2MetadataClient16GetCurrentRegionB5cxx11Ev+0x34c) [0x7faf345e44]
[4]/mnt/plugin/vigilcamera/lib/libaws-cpp-sdk-core.so(_ZN3Aws6Client19ClientConfigurationC1Ev+0x228) [0x7faf398318]
[5]./VIGILCameraServer(_ZN15CMotionUploader15GetSecurityKeysEv+0x420) [0x5990b8]
[6]./VIGILCameraServer(_ZN15CMotionUploader18InitializeAWSS3EnvEv+0x3c4) [0x59a9e4]
[7]./VIGILCameraServer(_ZN15CMotionUploaderC1Ev+0x18c) [0x597774]
[8]./VIGILCameraServer(_ZN18CVIGILCameraServer12CreateServerEv+0xb88) [0x4e3440]
9]./VIGILCameraServer(_ZN18CVIGILCameraServer10InitializeEv+0x64) [0x4e21f4]

[gi12345]

Reproduced this way:

int main(int argc, char** argv)
{
    printf("Point qqqqqqqqqqqqqqqqqq1\n");
    Aws::SDKOptions k_awsSDKoptions;
    Aws::InitAPI(k_awsSDKoptions);
    printf("Point qqqqqqqqqqqqqqqqqqqqqq2\n");
    {
    Aws::Client::ClientConfiguration m_cfg;  //xxx-gi
    printf("Point qqqqqqqqqqqqqqqqqqqqqqq3\n");
    }

=========================================
Log output:

Point qqqqqqqqqqqqqqqqqq1
Point qqqqqqqqqqqqqqqqqqqqqq2
Segmentation fault
#

=========================================
Obtained 8 stack frames:

[0]./VIGILCameraServer(_Z4dumpi+0x28) [0x4e51f8]
[1]linux-vdso.so.1(__kernel_rt_sigreturn+0) [0x7fa30885e4]
[2]/mnt/plugin/vigilcamera/lib/libaws-cpp-sdk-core.so(_ZNK3Aws8Internal21AWSHttpResourceClient34GetResourceWithAWSWebServiceResultB5cxx11ERKSt10shared_ptrINS_4Http11HttpRequestEE+0xa4) [0x7fa288cd7c]
[3]/mnt/plugin/vigilcamera/lib/libaws-cpp-sdk-core.so(_ZNK3Aws8Internal17EC2MetadataClient16GetCurrentRegionB5cxx11Ev+0x34c) [0x7fa2888e44]
[4]/mnt/plugin/vigilcamera/lib/libaws-cpp-sdk-core.so(_ZN3Aws6Client19ClientConfigurationC1Ev+0x228) [0x7fa28db318]
[5]./VIGILCameraServer(main+0x5b4) [0x4e5f24]
[6]/lib/libc.so.6(__libc_start_main+0xe8) [0x7fa1b30d18]
[7]./VIGILCameraServer() [0x4da934]

==========================================
Question: Could it be that core.so depends on some .so library which it cannot locate?
We are running on Ubuntu20-aarm64. SDK was built on EC2 with Ubuntu20-aarm64.