Nested signal handling leads to excessive slowdown of `sig_occurred()`

[user202729]

From sagemath/sage#39224

SageMath session:

sage: %%cython
....: from cysignals.signals cimport sig_on, sig_off, sig_occurred
....: from libc.stdlib cimport abort
....: def f():
....:     try:
....:         sig_on()
....:         abort()
....:         sig_off()
....:     except:
....:         raise ValueError
....: def g():
....:     for i in range(50):
....:         ignore = sig_occurred()
sage: %time g()  # fast
CPU times: user 4 μs, sys: 0 ns, total: 4 μs
Wall time: 6.44 μs
sage: f()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File /tmp/tmpgqxu9lji/spyx/_tmp_tmpfkjflhcs_tmp_f70vs9fj_pyx/_tmp_tmpfkjflhcs_tmp_f70vs9fj_pyx_0.pyx:5, in _tmp_tmpfkjflhcs_tmp_f70vs9fj_pyx_0.f()
      4 try:
----> 5     sig_on()
      6     abort()

RuntimeError: Aborted

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[3], line 1
----> 1 f()

File /tmp/tmpgqxu9lji/spyx/_tmp_tmpfkjflhcs_tmp_f70vs9fj_pyx/_tmp_tmpfkjflhcs_tmp_f70vs9fj_pyx_0.pyx:9, in _tmp_tmpfkjflhcs_tmp_f70vs9fj_pyx_0.f()
      7         sig_off()
      8     except:
----> 9         raise ValueError
     10 def g():
     11     for i in range(50):

ValueError: 
sage: %time g()  # slow
CPU times: user 3.84 s, sys: 133 μs, total: 3.84 s
Wall time: 3.85 s
sage: raise RuntimeError
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[5], line 1
----> 1 raise RuntimeError

RuntimeError: 
sage: %time g()  # fast again
CPU times: user 79.9 ms, sys: 0 ns, total: 79.9 ms
Wall time: 79.9 ms

Note the invocation of g() that is marked # slow.

[user202729]

I guess recursive .__context__ can be accessed. (.__cause__ is the same as .__context__ in the case of raise ValueError from e, but in nested exception then .__cause__ is None but .__context__ is not.)

That said, is there any cleaner way than the current (fragile) method of using reference counting and run the garbage collector? It looks like it will only be a matter of time until someone comes across a case where the garbage collector is run too many times consecutively which leads to unexplained slowdown.

[user202729]

Actually nested signal handling is not needed, you might as well do something more innocuous such as

%%cython
from libc.stdlib cimport abort
from cysignals.signals cimport sig_on
from cysignals.tests import print_sig_occurred
from sage.all import Integer

def f():
    a = [Integer(i) for i in range(100)]
    try:
        sig_on()
        abort()
    except RuntimeError:
        del a

But sig_occurred is introduced for a single purpose in SageMath sagemath/sage#24986 .

[user202729]

How about we throttle the garbage collector to run at most e.g. 10% of the (wall) time? That way, worst case something heavy is done in a place where sig_occurred() must honestly return nonnull and a lot of integers are being destroyed, there's ("only") 10% slowdown. And maybe a user warning.

[tornaria]

Here's a way to reproduce something what appears to be the root cause of this in python, using cysignals 1.11.4:

$ python
Python 3.12.7 (main, Oct  3 2024, 14:05:50) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cysignals.tests import test_abort, print_sig_occurred
>>> try:
...     test_abort()
... except:
...     raise ValueError
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "src/cysignals/tests.pyx", line 643, in cysignals.tests.test_abort
RuntimeError: Aborted

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
ValueError
>>> print_sig_occurred()
RuntimeError: Aborted

Clearly, by the time I typed print_sig_occurred(), the exception had long been handled and gone.

This also seems to be the cause of sagemath/sage#38749 (comment)

Using python 3.13.1 gives the same.

[tornaria]

No nesting of exceptions is necessary:

$ sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 10.5, Release Date: 2024-12-04                    │
│ Using Python 3.13.1. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
sage: from cysignals.tests import test_abort, print_sig_occurred
sage: test_abort()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[2], line 1
----> 1 test_abort()

File tests.pyx:634, in cysignals.tests.test_abort()

RuntimeError: Aborted
sage: raise ValueError
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[3], line 1
----> 1 raise ValueError

ValueError: 
sage: %time for _ in range(50): print_sig_occurred()
RuntimeError: Aborted
RuntimeError: Aborted
[...]
RuntimeError: Aborted
RuntimeError: Aborted
CPU times: user 2.04 s, sys: 617 μs, total: 2.04 s
Wall time: 2.05 s

Indeed, test_abort() raises the exception and leaves it at cysigs.exc_value. Doing raise ValueError before calling sig_occurred() hides it (the refcount is 2 but it's not in sys.last_value).

Does not happen with pure python, but it does happen with ipython:

$ ipython
Python 3.13.1 (main, Dec 14 2024, 12:44:03) [GCC 13.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.31.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from cysignals.tests import test_abort, print_sig_occurred

In [2]: test_abort()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[2], line 1
----> 1 test_abort()

File tests.pyx:634, in cysignals.tests.test_abort()

RuntimeError: Aborted

In [3]: raise ValueError
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[3], line 1
----> 1 raise ValueError

ValueError: 

In [4]: %time for _ in range(50): print_sig_occurred()
RuntimeError: Aborted
RuntimeError: Aborted
[...]
RuntimeError: Aborted
RuntimeError: Aborted
CPU times: user 672 ms, sys: 1.12 ms, total: 673 ms
Wall time: 676 ms

I suppose sage / ipython is storing the exception somewhere, which keeps the refcount from reaching 1.

[tornaria]

Here is a PoC fix:

--- a/src/cysignals/signals.pyx
+++ b/src/cysignals/signals.pyx
@@ -28,6 +28,7 @@ from libc.stdio cimport freopen, stdin
 from cpython.ref cimport Py_XINCREF, Py_CLEAR
 from cpython.exc cimport (PyErr_Occurred, PyErr_NormalizeException,
         PyErr_Fetch, PyErr_Restore)
+from cpython.exc cimport PyErr_GetRaisedException, PyErr_SetRaisedException
 from cpython.version cimport PY_MAJOR_VERSION
 
 cimport cython
@@ -360,6 +361,15 @@ cdef void verify_exc_value() noexcept:
     Check that ``cysigs.exc_value`` is still the exception being raised.
     Clear ``cysigs.exc_value`` if not.
     """
+    cdef PyObject * raised = PyErr_GetRaisedException()
+
+    if cysigs.exc_value is not raised:
+        Py_CLEAR(cysigs.exc_value)
+
+    PyErr_SetRaisedException(raised)
+
+    return
+
     if cysigs.exc_value.ob_refcnt == 1:
         # No other references => exception is certainly gone
         Py_CLEAR(cysigs.exc_value)

(note: as is, this requires python 3.12+ and a few tests fail)

BEFORE:

$ sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 10.5, Release Date: 2024-12-04                    │
│ Using Python 3.13.1. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
sage: R.<x> = Zmod(4)[] ; f = x^2 + 3*x + 1
sage: %time str(f)
CPU times: user 838 μs, sys: 0 ns, total: 838 μs
Wall time: 847 μs
'x^2 + 3*x + 1'
sage: from cysignals.tests import test_abort
sage: test_abort()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[4], line 1
----> 1 test_abort()

File tests.pyx:634, in cysignals.tests.test_abort()

RuntimeError: Aborted
sage: raise ValueError
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[5], line 1
----> 1 raise ValueError

ValueError: 
sage: %time str(f)
CPU times: user 711 ms, sys: 0 ns, total: 711 ms
Wall time: 713 ms
'x^2 + 3*x + 1'

AFTER:

$ sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 10.5, Release Date: 2024-12-04                    │
│ Using Python 3.13.1. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
sage: R.<x> = Zmod(4)[] ; f = x^2 + 3*x + 1
sage: %time str(f)
CPU times: user 841 μs, sys: 0 ns, total: 841 μs
Wall time: 847 μs
'x^2 + 3*x + 1'
sage: from cysignals.tests import test_abort
sage: test_abort()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[4], line 1
----> 1 test_abort()

File tests.pyx:634, in cysignals.tests.test_abort()

RuntimeError: Aborted
sage: raise ValueError
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[5], line 1
----> 1 raise ValueError

ValueError: 
sage: %time str(f)
CPU times: user 78 μs, sys: 13 μs, total: 91 μs
Wall time: 93.5 μs
'x^2 + 3*x + 1'

[user202729]

The problem with your fix is it negates the whole point of sig_occurred.

The reason why it's introduced is the following: (test with an environment with ipython setuptools cython)

%load_ext cython
##
%%cython
from cysignals.signals cimport sig_on, sig_off, sig_occurred
from cysignals.tests import print_sig_occurred
from libc.stdlib cimport abort

cdef class Integer:
    def __dealloc__(self):
        print_sig_occurred()
        print("deleted")

#class Integer:
#   def __del__(self):
#       print_sig_occurred()
#       print("deleted")

def mutate(a):
    abort()

def f1():
    a = Integer()
    sig_on()
    mutate(a)
    sig_off()

def g1():
    print("before call f1")
    f1()
    print("after call f1")
    # CANNOT print "no current exception" here

g1()

in SageMath, if mutate(a) is interrupted with e.g. a ctrl-C, a might be left in an inconsistent state, so fast_tp_dealloc checks for whether a signal occurred to avoid destroying a.

Of course, the solution implemented in SageMath is suboptimal in that the optimal way is to do for example change f1 to for example

def f1():
    a = Integer()
    try:
        sig_on()
        mutate(a.value)
        sig_off()
    except:
        a.value = None  # avoid calling mpz_clear(a.value) when a is deleted
        raise

but since there's a lot of usages of Integer in SageMath code base this may be more difficult than it sounds.

Of course in the long term we prefer this solution… but it is really hard to make sure it is bug-free while doing the migration.

---

Alternatively (potentially faster than the above solution because it avoids exception handler, and it's also shorter):

def f1():
    a = Integer()
    sig_on()
    cdef mpz a1 = a.value
    a.value = None  # if interrupted here, a.value is None so destructor is harmless
    mutate(a1)
    a.value = a1
    sig_off()

or put into helper function (then there's only 2 extra lines):

cdef take_mpz(a: Integer):
    """
    This function must be used before calling a mpz function that might be interrupted
    """
    cdef mpz a1 = a.value
    a.value = None
    return a1

def f1():
    a = Integer()
    sig_on()
    cdef mpz a1 = take_mpz(a)
    mutate(a1)
    a.value = a1
    sig_off()

[tornaria]

I understand what you mean. But how are we sure that every integer that was used since sig_on() will be deallocated before the exception is handled? This is what I meant by "scope" before: when should sig_occurred() stop reporting an exception has happened?

def f1():
    a = Integer()
    sig_on()
    mutate(a)
    sig_off()

In this case, I think cython will ensure a is deallocated before the function ends. But what if a came from outside? For example:

def f2(a):
    sig_on()
    mutate(a)
    sig_off()

def f1():
    a = Integer()
    try:
        f2(a)
    except:
        pass

[user202729]

• See the original issue and how difficult it was to debug it Inconsistent mpz_t state after interrupted sig_realloc() sage#24986
• SageMath Integer objects are supposed to be immutable. Any function that mutates Integer objects must be an internal function (and the Integer object likely something it has just created anew), so your case is excessively unlikely.
• Yes indeed, in your case the sig_occurred() fails; but according to the original issue above, the sig_occurred() is at most an ugly patch, and they just try their best effort there.

tl;dr… I guess we can just go with implementing the proper solution if you want. But:

• Not sure how large of a change it would need on SageMath's code base.
• Not sure how to test it even (actually looks like Windows is supported in GitHub Actions and there's https://github.com/marketplace/actions/install-cygwin … that might work partially)

[user202729]

The first bottleneck is there are so many occurrences of it. This is a conservative estimate (though in some cases the function call does not actually mutate the first argument):

[src]$ rg '\bmpz\w*\(\w*\.value' | wc -l
748