Segmentation fault when creating an `optiga_cmd`

[awidegreen]

Prerequisites

Can you reproduce the problem reliably?
Yes

Did you check current release notes for known issues?
Yes

If this is not the latest release, have you checked newer releases?
Yes

Description:

We ran into an issue in the latest version of the libTrustM with pal/linux running a relatively 'weak' SoC. We are getting a segmentation faults during startup of our application:

trustm_lib/extras/pal/linux/pal_os_event.c:129 pal_os_event_create: >
trustm_lib/extras/pal/linux/pal_os_event.c:67 pal_os_event_start: >
trustm_lib/extras/pal/linux/pal_os_event.c:196 pal_os_event_register_callback_oneshot: >
trustm_lib/extras/pal/linux/pal_os_event.c:52 handler: >
trustm_lib/extras/pal/linux/pal_os_event.c:163 pal_os_event_trigger_registered_callback: >
trustm_lib/extras/pal/linux/pal_os_event.c:196 pal_os_event_register_callback_oneshot: >
Segmentation fault (core dumped)
[exec] application (248:255) stopped by itself with exit code 139

Here is the backtrace

#0  0xb6f36320 in pal_os_event_register_callback_oneshot () from /nix/store/fbcan7yvy352hgrvqc6amspl73kkdqyl-linux-optiga-trust-m-armv7a-unknown-linux-gnueabihf-1.0.0-rco0/lib/libtrustm.so
#1  0xb6f36154 in pal_os_event_trigger_registered_callback ()
   from /nix/store/fbcan7yvy352hgrvqc6amspl73kkdqyl-linux-optiga-trust-m-armv7a-unknown-linux-gnueabihf-1.0.0-rco0/lib/libtrustm.so
#2  0xb6f36260 in handler () from /nix/store/fbcan7yvy352hgrvqc6amspl73kkdqyl-linux-optiga-trust-m-armv7a-unknown-linux-gnueabihf-1.0.0-rco0/lib/libtrustm.so
#3  <signal handler called>
#4  0xb6d07a68 in memcpy () from /nix/store/03xppsv10j3p4rp6bws8w9lx6dz1sydq-glibc-armv7a-unknown-linux-gnueabihf-2.35-224/lib/libc.so.6
#5  0xb6ce23e4 in _IO_default_xsputn () from /nix/store/03xppsv10j3p4rp6bws8w9lx6dz1sydq-glibc-armv7a-unknown-linux-gnueabihf-2.35-224/lib/libc.so.6
#6  0xb6ccbd70 in __vfprintf_internal () from /nix/store/03xppsv10j3p4rp6bws8w9lx6dz1sydq-glibc-armv7a-unknown-linux-gnueabihf-2.35-224/lib/libc.so.6
#7  0xb6ccc2d4 in buffered_vfprintf () from /nix/store/03xppsv10j3p4rp6bws8w9lx6dz1sydq-glibc-armv7a-unknown-linux-gnueabihf-2.35-224/lib/libc.so.6
#8  0xb6d790b4 in __fprintf_chk () from /nix/store/03xppsv10j3p4rp6bws8w9lx6dz1sydq-glibc-armv7a-unknown-linux-gnueabihf-2.35-224/lib/libc.so.6
#9  0xb6f36390 in pal_os_event_register_callback_oneshot () from /nix/store/fbcan7yvy352hgrvqc6amspl73kkdqyl-linux-optiga-trust-m-armv7a-unknown-linux-gnueabihf-1.0.0-rco0/lib/libtrustm.so
#10 0xb6f36520 in pal_os_event_start () from /nix/store/fbcan7yvy352hgrvqc6amspl73kkdqyl-linux-optiga-trust-m-armv7a-unknown-linux-gnueabihf-1.0.0-rco0/lib/libtrustm.so
#11 0xb6f36640 in pal_os_event_create () from /nix/store/fbcan7yvy352hgrvqc6amspl73kkdqyl-linux-optiga-trust-m-armv7a-unknown-linux-gnueabihf-1.0.0-rco0/lib/libtrustm.so
#12 0xb6f40d28 in optiga_cmd_create () from /nix/store/fbcan7yvy352hgrvqc6amspl73kkdqyl-linux-optiga-trust-m-armv7a-unknown-linux-gnueabihf-1.0.0-rco0/lib/libtrustm.so
#13 0xb6f374c0 in optiga_util_create () from /nix/store/fbcan7yvy352hgrvqc6amspl73kkdqyl-linux-optiga-trust-m-armv7a-unknown-linux-gnueabihf-1.0.0-rco0/lib/libtrustm.so
#14 0x0051c598 in ?? ()

By adding additional prints we tracked the issue to

optiga-trust-m/extras/pal/linux/pal_os_event.c

Line 199 in f67ec1c

p_pal_os_event->callback_registered = callback;

as the passed in event p_pal_os_event is NULL. Backtracking the start up, we found that there is a race-condition in starting the "pal timer" and returning of the pal_os_event_create in optiga_cmd_create.

Looking the callstack trace from above, we can see that the pal_os_event_create function is never exited, however the 'oneshot' timer is already fired (see handler call) - the timer is obviously started in pal_os_event_create. Now, the 'registered callback' is going to be called back optiga_cmd_queue_scheduler which will in turn call pal_os_event_register_callback_oneshot again with an event. However, this particular pal_os_event is taken from the p_optiga_ctx->p_pal_os_event_ctx which should have set to the return value of pal_os_event_create from above. But as pal_os_event_create hasn't returned yet, p_optiga_ctx->p_pal_os_event_ctx is still NULL, calling pal_os_event_register_callback_oneshot with a NULL as the *p_pal_os_event parameter, leading to the seg fault.

Steps to Reproduce:

Expected Result:
No seg fault

Actual Result:
Seg fault

Frequency (1/1, 1/50, 1/100):
1/2

Build/Commit:

Release V5.3, f67ec1c

Target:

Optiga TrustM V3

Host OS and Version:

Linux

armv7a-unknown-linux-gnueabihf GCC 11.3

Environment

yocto linux 6.6.51-linux4microchip-2024.10

[aoune-ayoub]

Hi @awidegreen ,

Thank you for creating the issue.

To solve the issue, we need first to reproduce it. can you please tell us more what you meant with "relatively 'weak' SoC" ?

If my understanding is right, you are running a Linux custom build generated with Yocto. you used the pal/linux PAL and started facing this issue ?

Kind regards,
Ayoub

[awidegreen]

Yes, we are running a custom yocto Linux together with the (unmodified) pal/linux PAL. The SoC is a single Cortex-A5 at 500MHz on a custom PCB.

A work-around we found working for us is to return the static instance of the pal_os_event_0 before calling pal_os_event_create

        if (FALSE == me->p_optiga->instance_init_state) {
            // create pal os event
+          me->p_optiga->p_pal_os_event_ctx = pal_os_event_static_event();
            me->p_optiga->p_pal_os_event_ctx =
                pal_os_event_create(optiga_cmd_queue_scheduler, me->p_optiga);
            me->p_optiga->p_optiga_comms = optiga_comms_create(optiga_cmd_execute_handler, me);

where as pal_os_event_static_event() in pal_os_event.c does:

pal_os_event_t *pal_os_event_static_event() {
    return (&pal_os_event_0);
}

The is probably not the desired fix to the issue; but it does eliminate the reported race-condition.

(untested) One way for you to reproduce this, might be adding a "delay" before returning from pal_os_event_create(). This way, the timer is already start and will fire before exiting pal_os_event_create().

[aoune-ayoub]

Hi @awidegreen ,

Glad to hear that you have found a workaround and you are no longer blocked by this issue.

We will try to reproduce the issue (including trying to add the delay). Once a fix is found, we will add it during next release.

Kind regards,
Ayoub