[BUG]: SIGABRT in PyNotifier on sim exit

[auscompgeek]

Problem description

13:41:05:742 INFO    : robotpy             : Robot code exited
terminate called without an active exception
Fatal Python error: Aborted

Thread 0x00007fd9fa200740 (most recent call first):
  <no Python frame>

Extension modules: _cffi_backend, numpy._core._multiarray_umath, numpy.linalg._umath_linalg, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator (total: 12)

(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007fd9f9a80183 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78
#2  0x00007fd9f9a26f9e in __GI_raise (sig=6) at ../sysdeps/posix/raise.c:26
#3  <signal handler called>
#4  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#5  0x00007fd9f9a80183 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78
#6  0x00007fd9f9a26f9e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#7  0x00007fd9f9a0e942 in __GI_abort () at abort.c:79
#8  0x00007fd9eaa09da9 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95
#9  0x00007fd9eaa1bc2c in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
#10 0x00007fd9eaa09951 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58
#11 0x00007fd9eaa0986e in __cxxabiv1::__gxx_personality_v0 (version=<optimized out>, actions=10, exception_class=0, ue_header=0x7fd9867fcd30, context=0x7fd9867fb540) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:677
#12 0x00007fd9eb2b8ca0 in _Unwind_ForcedUnwind_Phase2 (exc=exc@entry=0x7fd9867fcd30, context=context@entry=0x7fd9867fb540, frames_p=frames_p@entry=0x7fd9867fb448) at ../../../libgcc/unwind.inc:183
#13 0x00007fd9eb2b9380 in _Unwind_ForcedUnwind (exc=0x7fd9867fcd30, stop=stop@entry=0x7fd9f9a86a60 <unwind_stop>, stop_argument=<optimized out>) at ../../../libgcc/unwind.inc:218
#14 0x00007fd9f9a86c24 in __GI___pthread_unwind (buf=<optimized out>) at unwind.c:130
#15 0x00007fd9f9a7f452 in __do_cancel () at ../sysdeps/nptl/pthreadP.h:271
#16 __GI___pthread_exit (value=value@entry=0x0) at pthread_exit.c:36
#17 0x00007fd9f9cd5e10 in PyThread_exit_thread () at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Python/thread_pthread.h:418
#18 0x00007fd9f9c07388 in take_gil (tstate=tstate@entry=0x7fd998010940) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Python/ceval_gil.c:308
#19 0x00007fd9f9d45d8f in _PyEval_AcquireLock (tstate=0x7fd998010940) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Python/ceval_gil.c:585
#20 _PyThreadState_Attach (tstate=0x7fd998010940) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Python/pystate.c:2089
#21 PyEval_RestoreThread (tstate=0x7fd998010940) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Python/ceval_gil.c:651
#22 0x00007fd9e887a88c in std::_Function_handler<void (), frc::PyNotifier::PyNotifier(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) [clone .cold] ()
   from /home/davo/dev/frc/thedropbears/pyreefscape/.venv/lib64/python3.13/site-packages/wpilib/_wpilib.cpython-313-x86_64-linux-gnu.so
#23 0x00007fd9e8b20bde in pybind11::cpp_function::initialize<std::function<void ()>&, void, , pybind11::return_value_policy>(std::function<void ()>&, void (*)(), pybind11::return_value_policy const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) () from /home/davo/dev/frc/thedropbears/pyreefscape/.venv/lib64/python3.13/site-packages/wpilib/_wpilib.cpython-313-x86_64-linux-gnu.so
#24 0x00007fd9e8897fce in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) () from /home/davo/dev/frc/thedropbears/pyreefscape/.venv/lib64/python3.13/site-packages/wpilib/_wpilib.cpython-313-x86_64-linux-gnu.so
#25 0x00007fd9f9d70394 in cfunction_call (func=0x7fd9e05661b0, args=0x7fd9fa10afc8 <_PyRuntime+88296>, kwargs=<optimized out>) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Objects/methodobject.c:540
#26 0x00007fd9f9e28186 in _PyObject_Call (tstate=0x7fd998010940, callable=0x7fd9e05661b0, args=0x7fd9fa10afc8 <_PyRuntime+88296>, kwargs=<optimized out>) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Objects/call.c:361
#27 0x00007fd9f9d52ec0 in PyObject_Call (callable=0x7fd9e05661b0, args=0x7fd9fa10afc8 <_PyRuntime+88296>, kwargs=0x7fd9b5196ac0) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Objects/call.c:373
#28 PyCFunction_Call (callable=0x7fd9e05661b0, args=0x7fd9fa10afc8 <_PyRuntime+88296>, kwargs=0x7fd9b5196ac0) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Objects/call.c:381
#29 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Python/generated_cases.c.h:1355
#30 0x00007fd9f9da78d2 in _PyEval_EvalFrame (tstate=0x7fd998010940, frame=<optimized out>, throwflag=0) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Include/internal/pycore_ceval.h:119
#31 _PyEval_Vector (tstate=0x7fd998010940, func=0x7fd9eb6e6520, locals=0x0, args=0x7fd9867fbe08, argcount=1, kwnames=0x0) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Python/ceval.c:1811
#32 _PyFunction_Vectorcall (func=0x7fd9eb6e6520, stack=0x7fd9867fbe08, nargsf=1, kwnames=0x0) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Objects/call.c:413
#33 _PyObject_VectorcallTstate (tstate=0x7fd998010940, callable=0x7fd9eb6e6520, args=0x7fd9867fbe08, nargsf=1, kwnames=0x0) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Include/internal/pycore_call.h:168
#34 method_vectorcall (method=<optimized out>, args=0x7fd9fa10afe0 <_PyRuntime+88320>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Objects/classobject.c:70
#35 0x00007fd9f9e9c9a7 in thread_run (boot_raw=0x7fd998010900) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Modules/_threadmodule.c:337
#36 0x00007fd9f9e0c35c in pythread_wrapper (arg=<optimized out>) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Python/thread_pthread.h:243
#37 0x00007fd9f9a7e168 in start_thread (arg=<optimized out>) at pthread_create.c:448
#38 0x00007fd9f9b0214c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Operating System

Linux

Installed Python Packages

Package                  Version
------------------------ -----------
attrs                    24.2.0
bcrypt                   4.2.0
cffi                     1.17.1
cryptography             43.0.3
flexcache                0.3
flexparser               0.4
hypothesis               6.119.3
iniconfig                2.0.0
mypy                     1.13.0
mypy-extensions          1.0.0
numpy                    2.1.3
opencv-python            4.10.0.84
packaging                23.2
paramiko                 3.5.0
phoenix6                 25.1.0
photonlibpy              2025.1.1rc1
pint                     0.24.4
pip                      24.2
platformdirs             4.3.6
pluggy                   1.5.0
pycparser                2.22
pyfrc                    2025.0.0
pynacl                   1.5.0
pynetconsole             2.0.4
pyntcore                 2025.2.1
pytest                   8.3.3
pytest-integration       0.2.3
pytest-reraise           2.1.2
robotpy                  2025.2.1.0
robotpy-apriltag         2025.2.1
robotpy-cli              2024.0.0
robotpy-cscore           2025.2.1
robotpy-ctre             2025.0.0
robotpy-hal              2025.2.1
robotpy-halsim-gui       2025.2.1
robotpy-installer        2025.0.0
robotpy-rev              2025.0.1
robotpy-wpilib-utilities 2025.0.0
robotpy-wpimath          2025.2.1
robotpy-wpinet           2025.2.1
robotpy-wpiutil          2025.2.1
setuptools               75.5.0
sleipnirgroup-choreolib  2025.0.2
sortedcontainers         2.4.0
tomli                    2.1.0
tomlkit                  0.13.2
typing-extensions        4.12.2
wpilib                   2025.2.1

Reproducible example code

# https://github.com/thedropbears/pyreefscape/tree/a132aaf44a8b8ffaf8ccdc57018335a1301ed9d9

uv run robotpy sim

[virtuald]

I was able to duplicate this and looked a bit at our notifier implementation. Maaaaaybe we could detect the interpreter shutdown and just hang the thread instead of dying, but I just tried that and it didn't seem to work.

A real fix is coming if python/cpython#129688 is merged and pybind11 is updated... but then that's a 3.14 only change. See also discussion at https://discuss.python.org/t/safely-using-the-c-api-when-python-might-shut-down/78850

[virtuald]

Looking a bit more, we might be able to avoid this with judicious use of a flag set by atexit, which we currently use in wpiutil for safethread.